Top Open-Source and Self-Hosted AI Gateways

What is an open-source AI gateway?

An open-source AI gateway — also called an LLM proxy or inference server — is software you install on your own infrastructure that provides an API endpoint for AI model inference. Instead of calling a commercial API provider like OpenAI, you serve models yourself. The gateway handles model loading, request routing, tokenization, batching, and often adds features like caching, rate limiting, logging, and multi-model support.

Most open-source gateways expose an OpenAI-compatible endpoint (POST /v1/chat/completions), so your application code doesn't need to change — you just point your base_url at your own server instead of api.openai.com. This means you control the data flow end-to-end: your prompts never leave your network, you set the retention policy, and you pay only for the infrastructure, not per-token margins.

The gateway ecosystem spans several categories. Some tools — like LiteLLM — are primarily routing layers: they don't run inference themselves but forward requests to multiple providers (OpenAI, Anthropic, local models, cloud providers). Others — like vLLM, Ollama, and TGI — are inference engines: they load models onto GPU memory and perform the actual computation. Understanding this distinction is the first step in choosing the right tool.

Why self-host? The trade-offs

Self-hosting an AI gateway isn't just a preference; for many teams, it's a requirement. The motivations generally fall into three categories:

Complete data control

When you self-host, your prompts, completions, and any intermediate data stay on your infrastructure. Nothing passes through a third-party API. For regulated industries — healthcare (HIPAA), finance (PCI-DSS), government (FedRAMP) — this is often non-negotiable. The CLOUD Act also means US cloud providers can be compelled to hand over data stored abroad. Read more about GDPR vs. the CLOUD Act and data sovereignty.

Cost predictability at scale

Per-token pricing from commercial providers can be unpredictable at high volume. If your application processes millions of tokens daily, the cumulative cost adds up. Self-hosting shifts the cost from variable (per-token) to fixed (infrastructure), which can be significantly cheaper at scale. See our analysis of self-hosting vs. API cost for the math.

Customization and fine-tuning

Self-hosting lets you load custom fine-tuned models, experiment with quantization, swap model architectures without waiting for a provider to support them, and run models with modified system prompts or specialized tool calling configurations.

The trade-offs are real: you're responsible for GPU provisioning (GPUs are expensive and supply-constrained), model downloads and updates, load balancing across instances, monitoring and alerting, incident response (what happens when a GPU runs out of memory at 2 AM), security patches, and uptime SLAs. For teams without ML infrastructure expertise, the operational burden can outweigh the benefits.

LiteLLM — the routing standard

LiteLLM is the most popular open-source LLM gateway by GitHub stars and community adoption. It's a Python library and proxy server that routes requests to 100+ providers — OpenAI, Anthropic, Azure, AWS Bedrock, Google Vertex AI, local models via Ollama, and many more. It's MIT-licensed, and its README lists adopters such as Stripe, Netflix, Greptile and OpenHands.

Key features:

• Multi-provider routing — one endpoint, many backends. Define fallback chains so if your primary provider is down, requests automatically route to alternatives.
• Logging and observability — built-in logging to LangFuse, Helicone, LangSmith, Datadog, and custom destinations. Track latency, token usage, and costs per model, per team, per project.
• Caching — in-memory, disk, Redis/Valkey, S3 or GCS buckets, plus semantic caching on Redis or Qdrant. Cache identical prompts to save cost and reduce latency.
• Rate limiting and budget control — per-user, per-team, and global rate limits with configurable budget caps.
• Fallbacks and retries — automatic retry on failures with configurable backoff and fallback providers.
• OpenAI-compatible API — drop-in replacement for OpenAI's SDK.

When to use LiteLLM: Your team calls multiple providers (OpenAI, Anthropic, local models) and wants a single abstraction layer with observability. LiteLLM doesn't run inference itself — it routes to other providers or inference engines. It's ideal as the control plane for a multi-provider strategy.

Limitations: LiteLLM requires you to manage your own infrastructure for self-hosting. The proxy server runs on your servers, and you're responsible for uptime, scaling, and security. If you're routing to US-based providers, your data still passes through US jurisdiction — LiteLLM doesn't solve the sovereignty question, it just centralizes the routing.

Ollama — local inference made simple

Ollama has become the go-to tool for local LLM inference on developer machines. A single command — ollama run llama3 — downloads a model, loads it into memory, and gives you an interactive chat. It supports GGUF models (a quantized format optimized for CPU inference), runs on Mac, Linux, and Windows, and exposes an OpenAI-compatible API on localhost:11434.

Key features:

• Dead simple setup — one command to download and run any model from the Ollama library.
• GGUF support — optimized quantization for CPU inference, making it practical to run 70B+ models on consumer hardware.
• Cross-platform — native builds for macOS (Apple Silicon optimized), Linux, and Windows.
• Modelfile system — define custom system prompts, parameters, and adapters with simple text files.
• OpenAI-compatible API — a native POST /api/chat endpoint, plus OpenAI-compatible endpoints under /v1 (such as /v1/chat/completions) for seamless SDK integration.

When to use Ollama: Local development, prototyping, and proof-of-concept. It's excellent for testing how different models handle your prompts, evaluating model quality, and building local AI tools. Many developers use Ollama during development and switch to a production inference engine or managed API for production.

Limitations: Ollama isn't designed for production API serving. It can spread a model across several GPUs and serve a few requests in parallel, but it lacks load balancing, auto-scaling and monitoring. It's a developer tool, not an inference platform. For production workloads, teams typically graduate to vLLM, SGLang, or a managed provider.

vLLM — production throughput

vLLM is the leading open-source inference engine for production deployments. Its key innovation — PagedAttention — solves the memory fragmentation problem in transformer inference, achieving up to 24x higher throughput than Hugging Face Transformers in its 2023 launch benchmarks. Built by a community of over 2,000 contributors, it powers inference for many commercial AI services.

Key features:

• PagedAttention — efficient memory management that dramatically improves throughput without sacrificing quality.
• Continuous batching — processes requests of different lengths together, maximizing GPU utilization.
• High throughput — among the highest throughput in public benchmarks for supported models.
• Multi-GPU support — tensor parallelism across multiple GPUs for large models.
• OpenAI-compatible API — built-in OpenAI-compatible serving endpoint.
• Wide model support — supports LLaMA, Mistral, Mixtral, Falcon, Qwen, and many more architectures.
• Structured generation — built-in support for JSON mode and grammar-constrained decoding.

When to use vLLM: You have GPU infrastructure and need maximum throughput for production inference. It's the gold standard for self-hosted inference serving when you're running models at scale. Companies with dedicated ML infrastructure teams typically choose vLLM for their production inference stack.

Limitations: vLLM requires GPU infrastructure (in practice NVIDIA, AMD or Intel GPUs, or TPUs via plugins; a CPU backend exists), ML infrastructure expertise, and ongoing operational maintenance. You're responsible for model selection, quantization, scaling, monitoring, incident response, and uptime. It doesn't include observability, caching, or rate limiting out of the box — you build those layers yourself or on top of a proxy like LiteLLM.

Hugging Face TGI — transformers at scale

Hugging Face TGI (Text Generation Inference) is the production inference server from the Hugging Face team. Note: its repository now states that TGI is in maintenance mode (minor fixes only) and recommends vLLM or SGLang for new deployments. It's optimized for transformer models and provides an OpenAI-compatible API endpoint. It's designed for serving large language models at scale with features like tensor parallelism, speculative decoding, and optimized attention kernels.

Key features:

• Transformer-optimized — built specifically for Hugging Face model format, with tight integration to the Hugging Face Hub.
• Speculative decoding — uses a smaller draft model to accelerate generation, improving throughput.
• Tensor parallelism — distribute inference across multiple GPUs.
• OpenAI-compatible API — serves as a drop-in replacement for OpenAI's chat completions endpoint.
• Production-ready — Docker images, health checks, metrics endpoints, and load balancer integration.

When to use TGI: You're already in the Hugging Face ecosystem and need production-grade inference serving. It's particularly strong for teams that fine-tune models on Hugging Face and want a seamless path from training to serving.

Limitations: Like vLLM, TGI requires GPU infrastructure and ML ops expertise. It's focused on text generation — if you need embedding, classification, or other model types, you need additional infrastructure. The tool is mature but has a narrower scope than xinference (which supports multiple model types) and a smaller community than vLLM.

xinference — multi-model orchestration

xinference (Xorbits Inference, developed by the Xorbits team under the xorbitsai organization) is a comprehensive inference platform that supports multiple model types — chat, embedding, re-ranker, audio, image, and vision — from a single deployment. It's designed for teams that need to serve heterogeneous models in production.

Key features:

• Multi-model support — chat completion, embeddings, re-ranking, audio, and vision models all in one platform.
• Distributed deployment — scale across multiple nodes with automatic model placement.
• OpenAI-compatible API — standard endpoints for chat completions and embeddings.
• Large model library — access to models from Hugging Face, ModelScope, and custom local models.
• Quantization support — GGUF, AWQ, GPTQ, and other quantization formats.

When to use xinference: Your application needs multiple model types (chat, embedding, re-ranking, audio) and you want them all served from a unified platform. It's particularly useful for RAG pipelines that need both a chat model and an embedding model on the same infrastructure.

Limitations: xinference is less widely adopted than vLLM or LiteLLM, so the community and ecosystem are smaller. In production it usually runs on GPUs, and it still carries operational overhead. Documentation and community support are growing but don't yet match the maturity of vLLM or the simplicity of Ollama.

Comparison matrix

When to choose managed over self-hosted

Self-hosting makes sense when you have specific requirements that a managed provider can't meet. But for many teams, a managed gateway is the more practical choice. Here's a framework for deciding:

Self-host when:

• You have ML infrastructure expertise — dedicated DevOps/ML engineers who can manage GPU clusters, model updates, and incident response.
• You need complete data isolation — regulatory requirements that data must never leave your network (beyond what EU sovereign infrastructure provides).
• You're running custom fine-tuned models — proprietary models that you control end-to-end.
• Your volume justifies the overhead — millions of tokens daily where the fixed infrastructure cost is lower than per-token pricing. Read about when self-hosting becomes cost-effective.
• You need specific hardware — specialized GPUs or TPUs that aren't available through managed providers.

Choose managed when:

• You want to ship faster — no GPU procurement, no model downloads, no infrastructure setup. One API key and you're making calls.
• You want predictable costs — prepaid billing, no automatic top-up, no surprise infrastructure bills: calls are refused once the balance reaches zero (a call already in flight can take it slightly below).
• You need multi-model access — access to a curated catalog of the latest open-source models without managing downloads and updates.
• Compliance matters but self-hosting is overkill — EU sovereign infrastructure (like Frontière AI's sovereign models) keeps inference with EU-headquartered providers, outside the CLOUD Act's direct reach, without the operational burden of managing GPUs.
• You want developer tools — public benchmarks, rankings, model comparison matrices, and MCP endpoints for AI agents come built-in.

The reality is that most teams don't need to self-host. Self-hosting is justified at high volume or when regulatory requirements demand it. For the majority of use cases — especially for European teams that care about data sovereignty — a managed gateway with EU-sovereign models addresses the jurisdiction question without the operational overhead.

Frontière AI — the managed EU-sovereign alternative

Frontière AI bridges the gap between self-hosted control and managed convenience. It gives you the same OpenAI-compatible developer experience as the open-source gateways above, but with the advantages of a managed service built for EU sovereignty:

• Same API, zero infrastructure — POST /v1/chat/completions, same request format, same streaming behavior. Change your base_url and you're done. No GPUs to provision, no models to download, no load balancers to configure.
• EU sovereign infrastructure — 18 of the 25 live models run on OVHcloud and Scaleway (both French, no US control), so their inference stays with EU-headquartered companies, outside the CLOUD Act's direct reach — something no open-source gateway running on US cloud infrastructure can guarantee on its own. The other 7 are fast-access models served through US companies (Modal, OpenRouter) and labeled as such. The gateway application itself runs on Vercel's Paris region, a US sub-processor listed on our privacy page.
• Curated model catalog — GLM-5.2, Qwen3.5 397B (multimodal), Qwen3 235B, DeepSeek V4 Flash, Llama 3.3 70B, Qwen3.6 35B, Mistral Small 3.2, and gpt-oss 120B. Every model shows its Frontière Verified checklist (six criteria) on /benchmarks, each criterion ticked only once its check has been run, recorded and passed.
• Prepaid billing — top up from €10, calls consume your balance. No monthly commitment, no automatic top-up, no surprise charges. See pricing for the full table.
• Developer infrastructure — MCP endpoint for AI agents, rankings, model comparison matrix, API documentation, and public benchmarks.
• No prompt storage at Frontière AI — your prompts and replies are processed and never stored on our side (billing keeps only the model, the token counts and the amount); on fast-access models, retention at the provider follows that provider's own policy.

If you're evaluating LiteLLM for multi-provider routing, Frontière AI can be one of your backends — LiteLLM routes to Frontière AI's OpenAI-compatible endpoint just like it routes to any other provider. If you're using Ollama for local development, you can switch to Frontière AI for production without changing your application code. If you were considering vLLM for production inference but don't have GPU infrastructure, Frontière AI serves a curated selection of open models without the infrastructure burden.

FAQ