Top Open-Source and Self-Hosted AI Gateways

What is an open-source AI gateway?

An open-source AI gateway — also called an LLM proxy or inference server — is software you install on your own infrastructure that provides an API endpoint for AI model inference. Instead of calling a commercial API provider like OpenAI, you serve models yourself. The gateway handles model loading, request routing, tokenization, batching, and often adds features like caching, rate limiting, logging, and multi-model support.

Most open-source gateways expose an OpenAI-compatible endpoint (POST /v1/chat/completions), so your application code doesn't need to change — you just point your base_url at your own server instead of api.openai.com. This means you control the data flow end-to-end: your prompts never leave your network, you set the retention policy, and you pay only for the infrastructure, not per-token margins.

The gateway ecosystem spans several categories. Some tools — like LiteLLM — are primarily routing layers: they don't run inference themselves but forward requests to multiple providers (OpenAI, Anthropic, local models, cloud providers). Others — like vLLM, Ollama, and TGI — are inference engines: they load models onto GPU memory and perform the actual computation. Understanding this distinction is the first step in choosing the right tool.

Why self-host? The trade-offs

Self-hosting an AI gateway isn't just a preference; for many teams, it's a requirement. The motivations generally fall into three categories:

Complete data control

When you self-host, your prompts, completions, and any intermediate data stay on your infrastructure. Nothing passes through a third-party API. For regulated industries — healthcare (HIPAA), finance (PCI-DSS), government (FedRAMP) — this is often non-negotiable. The CLOUD Act also means US cloud providers can be compelled to hand over data stored abroad. Read more about GDPR vs. the CLOUD Act and data sovereignty.

Cost predictability at scale

Per-token pricing from commercial providers can be unpredictable at high volume. If your application processes millions of tokens daily, the cumulative cost adds up. Self-hosting shifts the cost from variable (per-token) to fixed (infrastructure), which can be significantly cheaper at scale. See our analysis of self-hosting vs. API cost for the math.

Customization and fine-tuning

Self-hosting lets you load custom fine-tuned models, experiment with quantization, swap model architectures without waiting for a provider to support them, and run models with modified system prompts or specialized tool calling configurations.

The trade-offs are real: you're responsible for GPU provisioning (GPUs are expensive and supply-constrained), model downloads and updates, load balancing across instances, monitoring and alerting, incident response (what happens when a GPU runs out of memory at 2 AM), security patches, and uptime SLAs. For teams without ML infrastructure expertise, the operational burden can outweigh the benefits.

LiteLLM — the routing standard

LiteLLM is the most popular open-source LLM gateway by GitHub stars and community adoption. It's a Python library and proxy server that routes requests to 100+ providers — OpenAI, Anthropic, Azure, AWS Bedrock, Google Vertex AI, local models via Ollama, and many more. It's MIT-licensed and used by companies including Shopify, Snowflake, and Netflix.

Key features:

• Multi-provider routing — one endpoint, many backends. Define fallback chains so if your primary provider is down, requests automatically route to alternatives.
• Logging and observability — built-in logging to LangFuse, Helicone, LangSmith, Datadog, and custom destinations. Track latency, token usage, and costs per model, per team, per project.
• Caching — Redis, MemGPT, and local disk caching. Cache identical prompts to save cost and reduce latency.
• Rate limiting and budget control — per-user, per-team, and global rate limits with configurable budget caps.
• Fallbacks and retries — automatic retry on failures with configurable backoff and fallback providers.
• OpenAI-compatible API — drop-in replacement for OpenAI's SDK.

When to use LiteLLM: Your team calls multiple providers (OpenAI, Anthropic, local models) and wants a single abstraction layer with observability. LiteLLM doesn't run inference itself — it routes to other providers or inference engines. It's ideal as the control plane for a multi-provider strategy.

Limitations: LiteLLM requires you to manage your own infrastructure for self-hosting. The proxy server runs on your servers, and you're responsible for uptime, scaling, and security. If you're routing to US-based providers, your data still passes through US jurisdiction — LiteLLM doesn't solve the sovereignty question, it just centralizes the routing.

Ollama — local inference made simple

Ollama has become the go-to tool for local LLM inference on developer machines. A single command — ollama run llama3 — downloads a model, loads it into memory, and gives you an interactive chat. It supports GGUF models (a quantized format optimized for CPU inference), runs on Mac, Linux, and Windows, and exposes an OpenAI-compatible API on localhost:11434.

Key features:

• Dead simple setup — one command to download and run any model from the Ollama library.
• GGUF support — optimized quantization for CPU inference, making it practical to run 70B+ models on consumer hardware.
• Cross-platform — native builds for macOS (Apple Silicon optimized), Linux, and Windows.
• Modelfile system — define custom system prompts, parameters, and adapters with simple text files.
• OpenAI-compatible API — POST /api/chat and OpenAI-compatible endpoints for seamless SDK integration.

When to use Ollama: Local development, prototyping, and proof-of-concept. It's excellent for testing how different models handle your prompts, evaluating model quality, and building local AI tools. Many developers use Ollama during development and switch to a production inference engine or managed API for production.

Limitations: Ollama isn't designed for production API serving. It lacks features like multi-GPU support, load balancing, auto-scaling, request batching, and monitoring. It's a developer tool, not an inference platform. For production workloads, teams typically graduate to vLLM, TGI, or a managed provider.

vLLM — production throughput

vLLM is the leading open-source inference engine for production deployments. Its key innovation — PagedAttention — solves the memory fragmentation problem in transformer inference, achieving up to 24x higher throughput than prior systems. It's used by thousands of companies and powers inference for many commercial AI services.

Key features:

• PagedAttention — efficient memory management that dramatically improves throughput without sacrificing quality.
• Continuous batching — processes requests of different lengths together, maximizing GPU utilization.
• High throughput — consistently the highest throughput in benchmarks for supported models.
• Multi-GPU support — tensor parallelism across multiple GPUs for large models.
• OpenAI-compatible API — built-in OpenAI-compatible serving endpoint.
• Wide model support — supports LLaMA, Mistral, Mixtral, Falcon, Qwen, and many more architectures.
• Structured generation — built-in support for JSON mode and grammar-constrained decoding.

When to use vLLM: You have GPU infrastructure and need maximum throughput for production inference. It's the gold standard for self-hosted inference serving when you're running models at scale. Companies with dedicated ML infrastructure teams typically choose vLLM for their production inference stack.

Limitations: vLLM requires GPU infrastructure (NVIDIA GPUs specifically), ML infrastructure expertise, and ongoing operational maintenance. You're responsible for model selection, quantization, scaling, monitoring, incident response, and uptime. It doesn't include observability, caching, or rate limiting out of the box — you build those layers yourself or on top of a proxy like LiteLLM.

Hugging Face TGI — transformers at scale

Hugging Face TGI (Text Generation Inference) is the production inference server from the Hugging Face team. It's optimized for transformer models and provides an OpenAI-compatible API endpoint. It's designed for serving large language models at scale with features like tensor parallelism, speculative decoding, and optimized attention kernels.

Key features:

• Transformer-optimized — built specifically for Hugging Face model format, with tight integration to the Hugging Face Hub.
• Speculative decoding — uses a smaller draft model to accelerate generation, improving throughput.
• Tensor parallelism — distribute inference across multiple GPUs.
• OpenAI-compatible API — serves as a drop-in replacement for OpenAI's chat completions endpoint.
• Production-ready — Docker images, health checks, metrics endpoints, and load balancer integration.

When to use TGI: You're already in the Hugging Face ecosystem and need production-grade inference serving. It's particularly strong for teams that fine-tune models on Hugging Face and want a seamless path from training to serving.

Limitations: Like vLLM, TGI requires GPU infrastructure and ML ops expertise. It's focused on text generation — if you need embedding, classification, or other model types, you need additional infrastructure. The tool is mature but has a narrower scope than xinference (which supports multiple model types) and a smaller community than vLLM.

xinference — multi-model orchestration

xinference (by the open-source team at SenseTime, now under the xorbitsai organization) is a comprehensive inference platform that supports multiple model types — chat, embedding, re-ranker, audio, image, and vision — from a single deployment. It's designed for teams that need to serve heterogeneous models in production.

Key features:

• Multi-model support — chat completion, embeddings, re-ranking, audio, and vision models all in one platform.
• Distributed deployment — scale across multiple nodes with automatic model placement.
• OpenAI-compatible API — standard endpoints for chat completions and embeddings.
• Large model library — access to models from Hugging Face, ModelScope, and custom local models.
• Quantization support — GGUF, AWQ, GPTQ, and other quantization formats.

When to use xinference: Your application needs multiple model types (chat, embedding, re-ranking, audio) and you want them all served from a unified platform. It's particularly useful for RAG pipelines that need both a chat model and an embedding model on the same infrastructure.

Limitations: xinference is less widely adopted than vLLM or LiteLLM, so the community and ecosystem are smaller. It still requires GPU infrastructure and operational overhead. Documentation and community support are growing but don't yet match the maturity of vLLM or the simplicity of Ollama.

Comparison matrix

	LiteLLM	Ollama	vLLM	TGI	xinference
Type	Routing proxy	Local inference	Inference engine	Inference server	Inference platform
Runs inference	No (routes)	Yes (CPU/GPU)	Yes (GPU)	Yes (GPU)	Yes (GPU)
Multi-provider routing	100+ providers	Local only	Single model	Single model	Multi-model
OpenAI-compatible API	Yes	Yes	Yes	Yes	Yes
Production ready	Yes	Dev / PoC	Yes (best throughput)	Yes	Yes
Multi-GPU	N/A	No	Yes	Yes	Yes (distributed)
Observability built-in	Yes (extensive)	Basic	Metrics only	Metrics only	Basic
Caching built-in	Yes	No	No	No	No
Rate limiting	Yes	No	No	No	No
GPU required	No	Optional	Yes (NVIDIA)	Yes (NVIDIA)	Yes
License	MIT	MIT	Apache-2.0	Apache-2.0	Apache-2.0

When to choose managed over self-hosted

Self-hosting makes sense when you have specific requirements that a managed provider can't meet. But for many teams, a managed gateway is the more practical choice. Here's a framework for deciding:

Self-host when:

• You have ML infrastructure expertise — dedicated DevOps/ML engineers who can manage GPU clusters, model updates, and incident response.
• You need complete data isolation — regulatory requirements that data must never leave your network (beyond what EU sovereign infrastructure provides).
• You're running custom fine-tuned models — proprietary models that you control end-to-end.
• Your volume justifies the overhead — millions of tokens daily where the fixed infrastructure cost is lower than per-token pricing. Read about when self-hosting becomes cost-effective.
• You need specific hardware — specialized GPUs or TPUs that aren't available through managed providers.

Choose managed when:

• You want to ship faster — no GPU procurement, no model downloads, no infrastructure setup. One API key and you're making calls.
• You want predictable costs — prepaid billing with a hard spending ceiling, no surprise infrastructure bills.
• You need multi-model access — access to a curated catalog of the latest open-source models without managing downloads and updates.
• Compliance matters but self-hosting is overkill — EU sovereign infrastructure (like Frontière AI) provides data residency and compliance without the operational burden of managing GPUs.
• You want developer tools — public benchmarks, rankings, model comparison matrices, and MCP endpoints for AI agents come built-in.

The reality is that most teams don't need to self-host. Self-hosting is justified at high volume or when regulatory requirements demand it. For the majority of use cases — especially for European teams that care about data sovereignty — a managed EU-sovereign gateway provides the compliance guarantees without the operational overhead.

Frontière AI — the managed EU-sovereign alternative

Frontière AI bridges the gap between self-hosted control and managed convenience. It gives you the same OpenAI-compatible developer experience as the open-source gateways above, but with the advantages of a managed service built for EU sovereignty:

• Same API, zero infrastructure — POST /v1/chat/completions, same request format, same streaming behavior. Change your base_url and you're done. No GPUs to provision, no models to download, no load balancers to configure.
• EU sovereign infrastructure — models run on OVHcloud and Scaleway (both French, no US ownership). Your data stays under European legal jurisdiction — something no open-source gateway running on US cloud infrastructure can guarantee on its own.
• Curated model catalog — GLM-5.2, Qwen3.5 397B (multimodal), Qwen3 235B, DeepSeek V4 Flash (1M context), Llama 3.3 70B, Qwen3.6 27B, Mistral Small 3.2, and gpt-oss 120B/20B. All models are verified with independent Frontière Verified scores.
• Prepaid billing — top up from €10, calls consume your balance. No monthly commitment, no surprise charges. See pricing for the full table.
• Developer infrastructure — MCP endpoint for AI agents, rankings, model comparison matrix, API documentation, and public benchmarks.
• No data logging for fine-tuning — your requests are processed, billed, and forgotten.

If you're evaluating LiteLLM for multi-provider routing, Frontière AI can be one of your backends — LiteLLM routes to Frontière AI's OpenAI-compatible endpoint just like it routes to any other provider. If you're using Ollama for local development, you can switch to Frontière AI for production without changing your application code. If you were considering vLLM for production inference but don't have GPU infrastructure, Frontière AI gives you the same model access without the infrastructure burden.

FAQ