Open Source AI Benchmark

Open Source AI Benchmark — Model Comparison 2026

A comparison of open source AI models by vendor-reported benchmark scores — each linked to its source, with its setting —, parameters, context window, license, and real EU pricing. Covers frontier models like Kimi K3, Qwen3.8, GLM-5.2, DeepSeek V4, and Llama 3.3 — and the models you can actually call through a sovereign EU API.

30

All Models

24

On Frontière AI

19

EU sovereign

21

Reasoning models

Benchmark scores at a glance

GPQA% correct, axis from 0 to 100
  1. Kimi K393.5
  2. Qwen3.8 Max92.6
  3. Qwen3.8 Flash91.7
  4. GLM-5.291.2
  5. DeepSeek V4.1 Flash90.9
  6. DeepSeek V4 Pro90.1
  7. Qwen3.8 27B89.2
  8. Qwen3.5 397B88.4
  9. Qwen3.6 27B87.8
  10. Nemotron 3 Ultra87.0
  11. Qwen3.6 35B A3B86.0
  12. Gemma 4 26B82.3
  13. Qwen3.5 9B81.7
  14. GPT-OSS 120B80.1
  15. Qwen3 235B77.5
  16. GPT-OSS 20B71.5
  17. Mistral Small 4 119B71.2
  18. Llama 3.3 70B50.5
  19. Qwen2.5-VL 72B49.0
  20. Mistral Small 3.2 24B46.13
  21. Gemma 3 27B42.4
  22. Apertus v1.5 70B32.1
MMLU% correct, axis from 0 to 100
  1. GPT-OSS 120B90.0
  2. Llama 3.3 70B86.0
  3. GPT-OSS 20B85.3
  4. Mistral Small 3.2 24B80.5
  5. Apertus v1.5 70B77.1
  6. Gemma 3 27B76.9
  7. Pixtral 12B69.2
LiveCodeBench% correct, axis from 0 to 100
  1. DeepSeek V4 Pro93.5
  2. Qwen3.8 Flash91.9
  3. Qwen3.8 27B90.3
  4. Nemotron 3 Ultra89.0
  5. Qwen3.6 27B83.9
  6. Qwen3.5 397B83.6
  7. Qwen3.6 35B A3B80.4
  8. Gemma 4 26B77.1
  9. Qwen3.5 9B65.6
  10. Mistral Small 4 119B63.6
  11. Qwen3 235B51.8
  12. Gemma 3 27B29.7
LMArenapreference rating, relative scale
  1. Kimi K31487
  2. GLM-5.31479
  3. GLM-5.21476
  4. GLM-5.3 Flash1474
  5. DeepSeek V4 Pro1458
  6. Qwen3.5 397B1442
  7. Qwen3.8 27B1438
  8. Gemma 4 26B1438
  9. Mistral Large 41432
  10. Nemotron 3 Ultra1426
  11. Qwen3 235B1422
  12. Gemma 3 27B1366
  13. Mistral Small 3.2 24B1356
  14. GPT-OSS 120B1352
  15. Llama 3.3 70B1318
  16. GPT-OSS 20B1317

One chart per benchmark, best score first; each value links to its source. Each vendor measures at its own setting (listed under the table), so bars are indicative rather than strictly comparable. LMArena has no natural zero: its bars are scaled to the spread shown, so their lengths are not proportional to the ratings. A model missing from a chart has no primary source for that score.

Capability vs. price

Models live on Frontière AI that have an LMArena score, by that score (a crowdsourced preference rating) and by price per 100k input tokens. Bigger bubbles = more parameters. Sovereign models on Frontière AI are highlighted.

155014751400132512500.000.100.210.310.420.52Price per 100k input tokens (€)LMArena scoreKimi K3 (2.8T parameters, 1M context)Kimi K3 (2.8T parameters, 1M context) LMArena: 1487 — entry “Kimi K3 (Max)”, ±5 Price / 100k: €0.4536 Params: 2800BDeepSeek V4 Pro (1.6T MoE)DeepSeek V4 Pro (1.6T MoE) LMArena: 1458 — entry “deepseek-v4-pro”, ±4, attributed by the leaderboard to the April 2026 release Price / 100k: €0.2453 Params: 1600BGLM-5.2 (Zhipu/Z.ai)GLM-5.2 (Zhipu/Z.ai) LMArena: 1476 — entry “glm-5.2-max”, ±4 Price / 100k: €0.3024 Params: 753BQwen3.5 397B (multimodal)Qwen3.5 397B (multimodal) LMArena: 1442 — entry “qwen3.5-397b-a17b”, ±3 Price / 100k: €0.1008 Params: 397BQwen3 235B (instruct)Qwen3 235B (instruct) LMArena: 1422 — entry “qwen3-235b-a22b-instruct-2507”, ±3 Price / 100k: €0.126 Params: 235BLlama 3.3 70BLlama 3.3 70B LMArena: 1318 — entry “llama-3.3-70b-instruct”, ±4 Price / 100k: €0.1512 Params: 70BGLM-5.3 (Zhipu/Z.ai)GLM-5.3 (Zhipu/Z.ai) LMArena: 1479 — entry “glm-5.3-max”, ±5 Price / 100k: €0.3024 Params: 753BGLM-5.3 Flash (Zhipu/Z.ai)GLM-5.3 Flash (Zhipu/Z.ai) LMArena: 1474 — entry “glm-5.3-flash”, ±5 Price / 100k: €0.0689 Params: 320BMistral Large 4 (preview, 512K context)Mistral Large 4 (preview, 512K context) LMArena: 1432 — entry “mistral-large-4”, ±7, listed as proprietary until the weights are published Price / 100k: €0.2514 Params: 1000BMistral Small 3.2 24BMistral Small 3.2 24B LMArena: 1356 — entry “mistral-small-2506”, ±5 Price / 100k: €0.0252 Params: 24BQwen3.8 27B (multimodal)Qwen3.8 27B (multimodal) LMArena: 1438 — entry “qwen3.8-27b”, ±5 Price / 100k: €0.0252 Params: 28BGemma 3 27B (Google)Gemma 3 27B (Google) LMArena: 1366 — entry “gemma-3-27b-it”, ±4 Price / 100k: €0.0135 Params: 27BGemma 4 26B (A4B MoE)Gemma 4 26B (A4B MoE) LMArena: 1438 — entry “gemma-4-26b-a4b”, ±7 Price / 100k: €0.0168 Params: 26BGPT-OSS 120B (OpenAI)GPT-OSS 120B (OpenAI) LMArena: 1352 — entry “gpt-oss-120b”, ±4 Price / 100k: €0.0252 Params: 117BSovereign (Frontière AI)Not EU sovereign

Models sorted by LMArena score by default; models without one follow, by GPQA. Click column headers to sort by MMLU, GPQA, LiveCodeBench, parameters, or LMArena. Every score links to its source, and “—” means no primary source reports that score for this exact model.

ModelLabParamsContextLicenseMMLUsetting per sourceGPQADiamond unless notedLiveCodeBenchversion per sourceLMArenatext, overallPrice / 100kJurisdictionVerified
🧠
Kimi2.8T/104B1MKimi K3 License—93.5—1487€0.4536Fast access3/6
🧠
GLM-5.3 (Zhipu/Z.ai)On Frontière AI
Zhipu / Z.ai753B512KGLM-5.3 License———1479€0.3024EU sovereign6/6
🧠
GLM-5.2 (Zhipu/Z.ai)On Frontière AI
Zhipu / Z.ai753B256kMIT—91.2—1476€0.3024EU sovereign6/6
🧠
Zhipu / Z.ai320B/18B1MMIT———1474€0.0689Fast access0/6
🧠
DeepSeek1.6T/49B1MMIT—90.193.51458€0.2453Fast access5/6
🧠
Qwen397B/17B250kApache-2.0—88.483.61442€0.1008EU sovereign6/6
🧠
Qwen3.8 27B (multimodal)On Frontière AI
Qwen27.8B256kApache-2.0—89.290.31438€0.0252EU sovereign6/6
🧠
Gemma 4 26B (A4B MoE)On Frontière AI
Google26B/4B256KApache-2.0—82.377.11438€0.0168EU sovereign6/6
🧠
Mistral AI1T/49B512k————1432€0.2514EU sovereign4/6
🧠
Nemotron 3 Ultra
NVIDIA550B/55B1MOpenMDW-1.1—87.089.01426N/A——
Qwen3 235B (instruct)On Frontière AI
Qwen235B/22B250kApache-2.0—77.551.81422€0.126EU sovereign5/6
Gemma 3 27B (Google)On Frontière AI
Google27B128kGemma Terms of Use76.942.429.71366€0.0135Fast access4/6
Mistral Small 3.2 24BOn Frontière AI
Mistral AI24B—Apache-2.080.546.13—1356€0.0252EU sovereign5/6
🧠
GPT-OSS 120B (OpenAI)On Frontière AI
OpenAI117B/5.1B—Apache-2.090.080.1—1352€0.0252EU sovereign6/6
Llama 3.3 70BOn Frontière AI
Meta70B—Llama 3.3 Community License86.050.5—1318€0.1512EU sovereign5/6
🧠
GPT-OSS 20B (OpenAI)Paused on Frontière AI
OpenAI21B/3.6B—Apache-2.085.371.5—1317N/AEU sovereign6/6
🧠
Qwen3.8 Max (2.4T, 1M context)Paused on Frontière AI
Qwen2.4T/95B1MQwen3.8-Max License—92.6——N/AFast access0/6
🧠
Qwen3.8 Flash (Qwen)On Frontière AI
Qwen125B/6B1MQwen Community License 1.0—91.791.9—€0.0252Fast access3/6
🧠
DeepSeek763B/8B prefill / 16B decode1MMIT—90.9——€0.1008EU sovereign6/6
🧠
Qwen3.6 27B (multimodal)Paused on Frontière AI
Qwen27.8B256kApache-2.0—87.883.9—N/AEU sovereign6/6
🧠
Qwen36B/3B256kApache-2.0—86.080.4—€0.042EU sovereign6/6
🧠
Qwen3.5 9B (fast, budget)Paused on Frontière AI
Qwen9.65B256kApache-2.0—81.765.6—N/AEU sovereign6/6
🧠
Mistral Small 4 119BOn Frontière AI
Mistral AI119B/6.5B256KApache-2.0—71.263.6—€0.0278EU sovereign5/6
Qwen2.5-VL 72B (vision)Paused on Frontière AI
Qwen73.4B32kQwen License Agreement—49.0——N/AEU sovereign5/6
🧠
Apertus72B100kApache-2.077.132.1——€0.1294EU sovereign5/6
🧠
DeepSeek V4 Flash 0731On Frontière AI
DeepSeek284B/13B256kMIT————€0.0672EU sovereign6/6
Qwen3 Coder 30BOn Frontière AI
Alibaba30.5B/3.3B256kApache-2.0————€0.0336EU sovereign6/6
Pixtral 12B (vision)On Frontière AI
Mistral AI12B128kApache-2.069.2———€0.0336EU sovereign5/6
BGE Multilingual Gemma2On Frontière AI
BAAI9.24B8,192Gemma Terms of Use————€0.0168EU sovereign—
Qwen3 Embedding 8BOn Frontière AI
Qwen8B40,960Apache-2.0————€0.0168EU sovereign—

Score sources

One primary source per score: the model's Hugging Face card, its technical report or its vendor's official announcement — and, for LMArena, the public leaderboard (text, overall, style control, snapshot of 2026-10-15). Every score was checked against its source between 2026-09-23 and 2026-10-20.

Sources and settings of every score

Benchmark methodology

Every score on this page is self-reported by the model's vendor — taken from its model card, technical report or official announcement — except LMArena, which comes from the public leaderboard. Frontière AI does not re-run these benchmarks. Vendors do not share one protocol (shots, chain of thought, reasoning effort, benchmark version), so each score states its setting and links to its source; scores measured under different settings are not strictly comparable. A score is shown only when a primary source reports it for the exact model listed (instruct rather than base model, the checkpoint Frontière AI serves); otherwise the cell shows “—”. We never estimate a score or borrow one from a sibling model.

MMLU (setting per source)

MMLU — 57 subjects covering STEM, humanities, law, medicine, and more. Measures general knowledge and reasoning across domains. Vendors report it 0-shot or 5-shot, with or without chain of thought; MMLU-Pro and MMLU-Redux are different benchmarks and are not shown in this column.

GPQA (Diamond unless noted)

GPQA — graduate-level science Q&A in physics, chemistry, and biology. Most vendors report the Diamond subset, the hardest one, validated by experts; when a source only writes “GPQA”, the score's note says so.

LiveCodeBench (version per source)

LiveCodeBench — coding problems collected continuously from competitive programming contests. Each version (v5, v6…) covers a different window of problems, so the version is given with every score.

LMArena (text, overall)

LMArena (formerly LMSYS Chatbot Arena) — crowdsourced blind A/B votes turned into a rating. We show the text leaderboard's overall score with style control, its default view, from the snapshot of 2026-10-15, with the leaderboard's own entry name. It measures which answers voters prefer, not accuracy, and it moves as votes come in.

Sources: Hugging Face model cards, vendors' technical reports and announcements, the LMArena leaderboard — one link per score, listed under the table.

How to read benchmark scores

Benchmark scores measure specific capabilities under controlled conditions. They do not predict real-world performance for your specific use case. A model that scores lower on MMLU may still be better for your workload due to context length, tool calling support, language coverage, or pricing. Use benchmarks as one input among many — and test models yourself with your actual prompts.

Frequently asked questions

What is the best open source AI model in 2026?
No single benchmark settles it, and every benchmark score on this page is the one the model's own vendor publishes, at its own setting (LMArena scores come from the public leaderboard). In this table, Kimi K3 has both the highest GPQA score (93.5) and the highest LMArena score (1487). Among the models in this table served by an EU-sovereign provider, GLM-5.2 has the highest GPQA score (91.2; LMArena 1476) and is served through Scaleway, from €0.3024 per 100k input tokens.
Can I self-host open source AI models?
Yes — all models listed here have downloadable weights. Requirements range from a single GPU for small models (Qwen3.5 9B, ~20 GB VRAM) to multi-GPU setups for frontier models (Kimi K3 needs a multi-node GPU cluster). Alternatively, Frontière AI provides instant API access to sovereign-hosted models starting at €10 prepaid.
Are open source AI models GDPR-compliant?
The model itself is never the issue — it's where it runs. A model hosted on OVHcloud or Scaleway in France stays under EU jurisdiction. The same model hosted on an US cloud provider is subject to the CLOUD Act. Frontière AI labels every model as 'EU sovereign' or 'fast access' so you know before you call.
Why do benchmark scores differ between sources?
Evaluators use different prompts, shot counts, chain-of-thought settings, reasoning-effort levels and benchmark versions — LiveCodeBench v5 and v6, for instance, cover different problems. Every score in this table is the one reported by the model's own vendor on its model card, technical report or announcement, or, for LMArena, by the public leaderboard; each links to that source and states its setting. Frontière AI does not re-run these benchmarks, so an independent evaluation of the same model can give a different figure, and two scores measured under different settings are not strictly comparable.
What's the difference between reasoning and non-reasoning models?
Reasoning models (marked with a thinking indicator) perform chain-of-thought internally before answering. They typically score higher on benchmarks involving math, code, and complex reasoning — but may be slower and more expensive due to extra tokens spent thinking. Non-reasoning models are faster and cheaper for straightforward tasks.
How does Frontière AI price open source models?
Frontière AI prices every model the same way: a published per-token rate for that model, plus one fixed markup. Displayed prices include our margin and VAT — the price shown is the price debited. Sovereign models start at €0.0168 per 100k input tokens (Gemma 4 26B); frontier models like GLM-5.2 start at €0.3024 per 100k input tokens. No subscription, purely prepaid.

Try these models yourself

Create an account and call any model in the catalog in minutes.

Create an accountModels →
Create an account