Benchmark numbers are the common currency of model comparison and are easy to misread. Knowing what each set contains, and what a given score means in context, is the difference between using them and being misled by them.
The standard set, and what it probes
These are the benchmarks you will see on nearly every model card. They measure genuinely different things, which is why models can be strong on one and weak on another.
| Benchmark | What it measures | Published reference points |
|---|---|---|
| MMLU | broad multiple-choice knowledge across many subjects | Chinchilla 67.5; DCLM 7B 64; Nemotron-CC 8B 70.3 vs Llama 3.1 8B 65.3 |
| GSM8K | grade-school multi-step arithmetic word problems | Tülu 3 8B 87.6; 70B 93.5 |
| MATH | competition-level mathematics | Tülu 3 8B 43.7; 70B 63.0 |
| IFEval | whether a model follows explicit, checkable instructions | Tülu 3 8B 82.4; 70B 83.2 |
| HumanEval | Python function synthesis from a docstring, pass@1 | phi-1 1.3B 50.6 |
| MBPP | basic Python programming problems | phi-1 1.3B 55.5 |
| AIME 24 / 25 | olympiad-level maths, now the headline reasoning test | OpenThinker3-7B 69.0 / 53.3 |
| GPQA-Diamond | graduate-level science questions resistant to search | OpenThinker3-7B 53.7 |
| LiveCodeBench | competitive programming, with contamination controls | OpenThinker3-7B 51.7 |
IFEval is the one to pay attention to
Most benchmarks test knowledge or reasoning. IFEval tests something more basic and often more commercially relevant: does the model do what it was told. Answer in exactly three sentences. Do not use commas. Reply in JSON. Include the word “therefore” twice.
Its virtue is that compliance is programmatically checkable, which makes the metric hard to argue with and the data cheap to generate. It is also why instruction-following shows up in RLVR mixes: Tülu 3’s verifiable prompt set is 14,973 instruction-following prompts alongside 7,473 GSM8K and 7,500 MATH.
If your application depends on structured output or format compliance — as most production integrations do — IFEval-style evaluation is a better predictor of whether a model will work for you than MMLU.
Contamination, and why scores drift upward
The central problem with public benchmarks is that they are public. Test items leak into pretraining corpora, and a model that has seen the answers scores well without the capability.
This is not hypothetical. The web-scale corpora are built from Common Crawl, which contains benchmark pages, discussion of benchmark questions, and repositories of worked solutions. Deduplication against test sets helps and is not complete.
Three practical defences. Prefer benchmarks with contamination controls — LiveCodeBench dates its problems specifically so you can evaluate only on items postdating a model's training. Watch for the signature of contamination: excellent benchmark scores alongside disappointing behaviour on your own similar-looking tasks. And keep a private evaluation set, which is the only defence that fully works.
The same logic applies to generated training data: if your training and evaluation data come from the same teacher model, your benchmark measures agreement with that teacher rather than correctness.
What public benchmarks cannot tell you
They are built to compare models on general capability. Your question is almost always narrower: will this model do my task on my inputs.
No public benchmark covers your document formats, your domain vocabulary, your users' phrasing, your latency budget or your failure costs. A model can lead every leaderboard and be the wrong choice for you, usually because your inputs are unlike anything in the benchmark suite.
So use the public numbers for what they are good at — shortlisting two or three candidates — and decide between them on your own evaluation set. That set is also the only thing that can tell you whether a fine-tune helped, which makes it the highest-value artefact in the whole pipeline.
Questions people ask
Which benchmarks should I report for a fine-tuned model?
The ones covering what you changed and the ones covering what you did not. A fine-tune that improves your task while degrading general capability is a common outcome, and reporting only the target metric hides it. Include at least one general benchmark as a regression check.
Is a 2-point benchmark difference meaningful?
Usually not on its own. Scores vary with prompt formatting, few-shot examples, decoding parameters and evaluation harness. Treat small gaps between models as noise unless the evaluation setup is identical, and be wary of comparisons assembled from different papers' reported figures.
How large should my own evaluation set be?
A few hundred hand-checked examples is enough to be useful and small enough to actually curate. Coverage of your real input distribution matters far more than size, and a set you have read every item of is worth more than a large one you have not.
Should I use a model as a judge?
It is practical and widely done — UltraFeedback uses GPT-4 ratings across four aspects. The limits are the judge's biases, a known preference for longer and more confident answers, and the circularity if the judge also generated your training data. Validate the judge against human ratings on a slice before trusting it at scale.