Integration guide

Run LiteLLM on EU-hosted open models

LiteLLM talks to any OpenAI-compatible API through its openai/ prefix, so Frontière AI plugs in as one more upstream: model openai/<model>, api_base pointing at our API, and your sk-front key. You keep everything LiteLLM gives you — virtual keys, budgets, fallbacks, logging — and the models behind it are open-weight ones, with those labeled EU sovereign served by European operators. Every snippet on this page was run against our gateway and the real providers behind it, with LiteLLM 1.105.0.

Tested on 2026-10-12 with LiteLLM 1.105.0: chat, streaming with usage, tool calling, embeddings, the proxy with cost tracking, the /v1/messages and /v1/responses bridge, and the routing headers. What it did not cover: a production deployment of the proxy with a database, keys and teams — that part is LiteLLM's own.

What this gives you

LiteLLM is the layer teams put in front of their model providers: one interface, one place for keys, budgets and fallbacks. Frontière AI is a provider in that picture — one API key that reaches a curated catalog of open-weight models (Qwen, DeepSeek, GLM, Mistral, gpt-oss and others), each served by a named provider and labeled before you call it: EU sovereign, or fast access when a US company serves it.

The two do not overlap. LiteLLM decides which model a request goes to and enforces your spending rules; Frontière AI decides which provider serves that model and tells you, on every response, which one did. If your LiteLLM already routes to OpenAI or Anthropic, adding Frontière AI gives the same proxy a European lane for the workloads your DPO cares about.

Before you start

• LiteLLM installed — pip install litellm for the SDK, pip install 'litellm[proxy]' for the proxy.
• A Frontière AI API key (dashboard → API keys) and a balance above zero: calls on an empty balance return 402.
• A model ID from our catalog. GET /models is public, so you can choose before you even have a key — and only its IDs are valid after the openai/ prefix.

1. Python SDK

Put openai/ in front of our model ID, and pass our API as api_base. The prefix is what tells LiteLLM to speak the OpenAI protocol to a base URL that is not OpenAI's; without it, LiteLLM cannot tell which provider you mean and refuses the call.

import os
import litellm

response = litellm.completion(
    model="openai/qwen3-235b",
    api_base="https://getfrontiereai.eu/api/v1",
    api_key=os.environ["FRONTIERE_KEY"],  # your Frontière AI key (dashboard → API keys)
    messages=[{"role": "user", "content": "Bonjour"}],
)
print(response.choices[0].message.content)

api_base stops at /api/v1. LiteLLM appends /chat/completions itself — add it yourself and the request lands on a path that does not exist.

2. LiteLLM Proxy

On the proxy, each Frontière AI model becomes one entry of model_list. model_name is the name your clients will use; litellm_params carries the same three settings as the SDK. Keep the key out of the file with the os.environ/ syntax.

model_list:
  - model_name: qwen3-235b
    litellm_params:
      model: openai/qwen3-235b
      api_base: https://getfrontiereai.eu/api/v1
      api_key: os.environ/FRONTIERE_KEY
    model_info:
      # our prices for this model, EUR per token, margin and VAT included
      input_cost_per_token: 0.00000126
      output_cost_per_token: 0.00000378
  # embedding model: same settings, mode embedding
  - model_name: bge-multilingual-gemma2
    litellm_params:
      model: openai/bge-multilingual-gemma2
      api_base: https://getfrontiereai.eu/api/v1
      api_key: os.environ/FRONTIERE_KEY
    model_info:
      mode: embedding
      input_cost_per_token: 0.000000168

litellm_settings:
  drop_params: true

drop_params: true makes LiteLLM drop parameters an upstream does not support instead of failing the request — useful when clients written for OpenAI send options an open-weight model ignores. The prices in model_info are computed from our catalog for you; the spend-tracking section explains them.

export FRONTIERE_KEY="sk-front-…"
# recent LiteLLM versions refuse to start without it
export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 24)"
litellm --config config.yaml --port 4000

Your clients then call the proxy, not us, with the proxy's key — the same OpenAI client, a different base URL:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:4000/v1", api_key=os.environ["LITELLM_MASTER_KEY"])
reply = client.chat.completions.create(
    model="qwen3-235b",
    messages=[{"role": "user", "content": "Bonjour"}],
)

Clients that speak /v1/messages or /v1/responses

Some clients do not call /chat/completions at all: Anthropic-style tools send /v1/messages, newer OpenAI tools send /v1/responses. Our API serves chat completions, so with the openai/ prefix the proxy forwards those two to a path we do not have and they fail with a 404. Declare the chat models with the custom_openai/ prefix instead — same api_base, same key — and LiteLLM translates both into chat completions before calling us.

model_list:
  - model_name: qwen3-235b
    litellm_params:
      model: custom_openai/qwen3-235b
      api_base: https://getfrontiereai.eu/api/v1
      api_key: os.environ/FRONTIERE_KEY

We tested it with a plain request on each route: an Anthropic-format /v1/messages call and a /v1/responses call both came back answered by our model. We have not run Claude Code or Codex end to end through it. Keep openai/ for embedding models — custom_openai/ does not handle embeddings.

3. Streaming and usage

Streaming works as with OpenAI. Our gateway always sends the token usage in the final chunk of a stream, but the LiteLLM SDK only passes it on when the call asks for it: without stream_options it was simply not in the chunks LiteLLM handed back. Ask for it explicitly if your code reads usage from a stream.

stream = litellm.completion(
    model="openai/qwen3-235b",
    api_base="https://getfrontiereai.eu/api/v1",
    api_key=os.environ["FRONTIERE_KEY"],
    messages=[{"role": "user", "content": "Bonjour"}],
    stream=True,
    stream_options={"include_usage": True},  # ask for usage explicitly: LiteLLM only passes it on when requested
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
    if getattr(chunk, "usage", None):
        print("\n", chunk.usage)

Behind the proxy, the same rule applies to your clients: they receive the usage chunk when they ask for it, or always if you set always_include_stream_usage: true under general_settings. Either way, billing on our side does not depend on it: we count the tokens from the provider's own report, whether or not your client reads them.

4. Tool calling

tools and tool_choice pass through LiteLLM and through our gateway untouched. In our test, a get_weather tool came back as a proper tool_calls entry, with finish_reason tool_calls and the arguments as valid JSON.

Whether a model emits tool calls at all depends on the model and its provider, so we measure it with one real call per model rather than assume it: 24 of the 24 live models measured so far returned a valid tool call. Agents built on LiteLLM should start from those.

See the full tool-calling matrix →

5. Embeddings

Same prefix, same api_base, through litellm.embedding — or a model_list entry marked mode: embedding on the proxy. Use one of the embedding models (bge-multilingual-gemma2 and qwen3-embedding-8b); a chat model ID on the embeddings endpoint answers 404 with an explicit message. With a chat model labeled EU sovereign, both halves of a retrieval pipeline then stay on European infrastructure, since every embedding model we serve is EU-hosted.

vectors = litellm.embedding(
    model="openai/bge-multilingual-gemma2",
    api_base="https://getfrontiereai.eu/api/v1",
    api_key=os.environ["FRONTIERE_KEY"],
    input=["bonjour", "hello"],
)

6. Spend tracking

LiteLLM ships a price list for the models it knows; ours are not in it, so out of the box it records our calls at zero cost. Tell it our prices in model_info — input_cost_per_token and output_cost_per_token — and its spend tracking, budgets and the x-litellm-response-cost header become exact.

The values in the config above are our current prices for those models, in euros per token, margin and VAT included — the amount actually debited from your balance. They are computed from our catalog when this page is built, so copy them from here rather than from an old config.

LiteLLM does not convert currencies: it multiplies tokens by the per-token cost you enter. With our prices entered, the spend it tracks is in euros — worth remembering if the same budget also covers models priced in dollars.

Who served each call

Every response from our gateway carries x-routed-provider (the provider that served the call) and x-sovereign (whether that provider is one we label EU sovereign). LiteLLM keeps them: on the proxy's response they come back prefixed llm_provider-, and in the SDK they sit in the response's hidden parameters.

# SDK — which provider served the call, and whether it is labeled EU sovereign
headers = response._hidden_params["additional_headers"]
headers["llm_provider-x-routed-provider"]   # e.g. "scaleway"
headers["llm_provider-x-sovereign"]         # "true"

# Proxy — the same two headers on its HTTP response
curl -si http://localhost:4000/v1/chat/completions … | grep llm_provider-x-

Log them next to LiteLLM's own request log and you have, per call, a record of which jurisdiction served it — the question an audit asks after the fact, and the one a blended gateway usually cannot answer.

What it changes, and what it doesn't

The inference moves to the provider hosting the model, and the model's label tells you which jurisdiction that is. LiteLLM itself stays wherever you deployed it: your proxy sees every prompt before we do, and its logs and database are yours to place. Self-hosted in Europe with a sovereign model, the whole path runs in Europe — our gateway itself runs in Paris on Vercel, a US company, with no prompt or response stored (our privacy page says so).

One more boundary worth stating to whoever signs off: if the same proxy also routes to US providers, LiteLLM's fallbacks can move a request from one to the other. Keep fallback chains for regulated workloads inside models labeled EU sovereign.

GDPR vs the CLOUD Act →

Troubleshooting

The errors below are the exact messages we got from LiteLLM 1.105.0:

Parameter names checked in LiteLLM's documentation on 2026-10-12: docs.litellm.ai — OpenAI-compatible endpoints · docs.litellm.ai — proxy config.yaml · docs.litellm.ai — custom pricing · docs.litellm.ai — streaming usage · docs.litellm.ai — drop_params · docs.litellm.ai — /v1/responses & /v1/messages passthrough · PyPI — litellm

FAQ