Open Source AI Benchmark

Open Source AI Benchmark — Model Comparison 2026

An independent comparison of open source AI models by benchmark scores, parameters, context window, license, and real EU pricing. Covers frontier models like Kimi K3, Qwen3.8, GLM-5.2, DeepSeek V4, and Llama 3.3 — and the models you can actually call through a sovereign EU API.

All Models

On Frontière AI

EU sovereign

Reasoning models

Benchmark scores at a glance

MMLU

GPQA

Sovereign (Frontière AI)

Not on Frontière AI

Kimi 2.8T

89.5

Qwen 2.4T

89.2

DeepSeek ~1.6T

88.1

GLM-5.2 (Zhipu/Z.ai)

87.6

NVIDIA ~692B

87.2

Qwen3.5 397B (multimodal)

86.4

Qwen3 235B (instruct)

84.7

DeepSeek V4 Flash 0731 (1M context)

83.2

Llama 3.3 70B

79.5

Qwen3.6 27B (multimodal)

78.9

Qwen3 Coder 30B

75.3

Qwen3 32B

76.8

Mistral Small 3.2 24B

74.2

Qwen3.5 9B (fast, budget)

69.4

OpenAI 120B

80.1

0255075100

Capability vs. price

How open source models compare by intelligence (Arena Elo) and cost per 100k input tokens on sovereign EU infrastructure. Bigger bubbles = more parameters. Sovereign models on Frontière AI are highlighted.

Models sorted by Arena Elo by default. Click column headers to sort by MMLU, GPQA, LiveCodeBench, parameters, or price. All scores come from public model cards, technical reports, and LMSYS Chatbot Arena.

Model	Lab	Params	Context	License	MMLU	GPQA	LiveCode	Arena Elo	Price / 100k	Jurisdiction	Verified
🧠 Kimi (2.8T)	Kimi	2.8T/~120B	256K	MIT	89.5	74.2	58.3	1342	N/A	—	—
🧠 Qwen (2.4T)	Qwen	2.4T/~95B	256K	Apache-2.0	89.2	72.8	56.1	1328	N/A	—	—
🧠 DeepSeek (~1.6T)	DeepSeek	~1.6T/~100B	128K	MIT	88.1	71.5	55.4	1310	N/A	—	—
🧠 GLM-5.2 (Zhipu/Z.ai)On Frontière AI	Zhipu / Z.ai	753B	128K	MIT	87.6	69.8	52.7	1295	€0.252	EU sovereign	6/6
NVIDIA (~692B)	NVIDIA	~692B/~60B	128K	Apache-2.0	87.2	68.5	48.9	1278	N/A	—	—
Qwen3.5 397B (multimodal)On Frontière AI	Qwen	403B/17B	128K	Apache-2.0	86.4	66.2	46.3	1260	€0.084	EU sovereign	6/6
Qwen3 235B (instruct)On Frontière AI	Qwen	235B/22B	128K	Apache-2.0	84.7	62.1	42.5	1230	€0.105	EU sovereign	0/6
🧠 DeepSeek V4 Flash 0731 (1M context)On Frontière AI	DeepSeek	158B/13B	1M	MIT	83.2	58.9	40.1	1210	€0.056	EU sovereign	6/6
Llama 3.3 70BOn Frontière AI	Meta	70B	128K	Llama Community	79.5	54.3	38.2	1180	€0.0898	EU sovereign	5/6
Qwen3.6 27B (multimodal)On Frontière AI	Qwen	27.8B	128K	Apache-2.0	78.9	52.1	35.7	1165	€0.0574	EU sovereign	6/6
Qwen3 Coder 30BOn Frontière AI	Alibaba	30.5B/3B	128K	Apache-2.0	75.3	48.6	44.2	1120	€0.0084	EU sovereign	6/6
Qwen3 32BOn Frontière AI	Alibaba	32.8B	4K	Apache-2.0	76.8	50.4	33.1	1140	€0.0112	EU sovereign	6/6
Mistral Small 3.2 24BOn Frontière AI	Mistral AI	24B	32K	Apache-2.0	74.2	46.8	30.5	1100	€0.0126	EU sovereign	6/6
Qwen3.5 9B (fast, budget)On Frontière AI	Qwen	9.65B	128K	Apache-2.0	69.4	40.2	25.8	1050	€0.014	EU sovereign	6/6
OpenAI (120B)	OpenAI	120B/~20B	128K	Apache-2.0	80.1	55.7	39.4	1190	N/A	—	—
BAAI (768M)On Frontière AI	BAAI	768M	8K	Apache-2.0	—	—	—	—	N/A	—	—
Qwen (8.2B)On Frontière AI	Qwen	8.2B	4K	Apache-2.0	—	—	—	—	N/A	—	—

Benchmark methodology

Scores are sourced from official model cards and technical reports. Where multiple reports exist, we use the highest independently verified figure. All benchmarks follow their official evaluation protocol.

MMLU (5-shot)

MMLU (5-shot) — 57 subjects covering STEM, humanities, law, medicine, and more. Measures general knowledge and reasoning across domains.

GPQA Diamond

GPQA Diamond — Graduate-level science Q&A in physics, chemistry, and biology. Only the hardest subset with expert agreement, considered the gold standard for scientific reasoning.

LiveCodeBench v5

LiveCodeBench v5 — Real-world coding tasks from live competitive programming platforms. Measures practical code generation ability.

LMSYS Chatbot Arena

LMSYS Chatbot Arena Elo — Crowdsourced blind A/B comparisons. The most trusted real-world performance signal, updated continuously. Higher Elo = better user-preferred responses.

Sources: Hugging Face model cards, official technical reports, LMSYS Chatbot Arena leaderboard.

How to read benchmark scores

Benchmark scores measure specific capabilities under controlled conditions. They do not predict real-world performance for your specific use case. A model that scores lower on MMLU may still be better for your workload due to context length, tool calling support, language coverage, or pricing. Use benchmarks as one input among many — and test models yourself with your actual prompts.

Frequently asked questions

What is the best open source AI model in 2026?

Kimi K3 and Qwen3.8 2.4T lead in raw intelligence (Arena Elo 1340+), followed closely by DeepSeek V4 Pro and GLM-5.2. For practical use, GLM-5.2 is the strongest model available through a sovereign EU API — scoring 87.6% on MMLU and 69.8% on GPQA Diamond, served through Scaleway starting at €0.25 per 100k input tokens.

Can I self-host open source AI models?

Yes — all models listed here have downloadable weights. Requirements range from a single GPU for small models (Qwen3.5 9B, ~20 GB VRAM) to multi-GPU setups for frontier models (Kimi K3 requires 8×H100 minimum). Alternatively, Frontière AI provides instant API access to sovereign-hosted models starting at €10 prepaid.

Are open source AI models GDPR-compliant?

The model itself is never the issue — it's where it runs. A model hosted on OVHcloud or Scaleway in France stays under EU jurisdiction. The same model hosted on an US cloud provider is subject to the CLOUD Act. Frontière AI labels every model as 'EU sovereign' or 'fast access' so you know before you call.

Why do benchmark scores differ between sources?

Different benchmark providers use different prompts, temperature settings, and evaluation splits. We source scores from official model cards and technical reports to ensure consistency. Even then, a model reported at 87.6% MMLU by its creator might score 86% in an independent evaluation — both are legitimate measurements of the same model.

What's the difference between reasoning and non-reasoning models?

Reasoning models (marked with a thinking indicator) perform chain-of-thought internally before answering. They typically score higher on benchmarks involving math, code, and complex reasoning — but may be slower and more expensive due to extra tokens spent thinking. Non-reasoning models are faster and cheaper for straightforward tasks.

How does Frontière AI price open source models?

Frontière AI applies a flat 40% margin on top of infrastructure costs. Sovereign providers (OVHcloud, Scaleway) are the cheapest — starting at €0.0045 per 100k input tokens for Qwen3.5 9B. Frontier models like GLM-5.2 start at €0.25 per 100k input tokens. No subscription, purely prepaid.

Try these models yourself

Create an account and call any model in the catalog in minutes.

Create an account Models →