Building a golden evaluation set

Build it before the training set

This ordering is the single most consequential process choice in a fine-tuning project, and the temptation to invert it is strong: training data feels like progress, evaluation feels like overhead.

But without a held-out set you cannot distinguish a better dataset from a luckier run. You cannot tell whether doubling your data helped. You cannot tell whether the model you shipped is better than the one before it. Every question becomes a matter of impression, and impressions on a handful of hand-typed prompts are reliably wrong.

Building it first has a second benefit: writing down what a correct output looks like forces you to define the task precisely, which usually changes what training data you decide to collect. Several teams discover at this point that they disagree internally about what the model should do.

What goes in it

Coverage of your real input distribution matters far more than size. A few hundred items you have read every one of beats several thousand you have not.

Sample from real traffic if you have it. If you do not, write items that reflect what you expect — and revisit the set once you do have traffic, because the gap between expected and actual inputs is usually large and instructive.

Deliberately include the categories that break things:

• Typical cases, in proportion to real frequency — the bulk of the set.
• The long tail: unusual phrasings, rare formats, edge cases. Over-represent these relative to their frequency, because they are where failures concentrate.
• Inputs that should be refused or escalated, if your application has any — otherwise you only ever measure compliance.
• Adversarial and malformed inputs: empty, truncated, wrong language, prompt-injection attempts, inputs in the wrong format.
• Cases where the correct answer is “I don't know” — the hardest thing to train and the easiest to forget to test.
• Known past failures. Every production incident should end with an item added here. This is how the set earns its keep over time.

Make the grading decidable

An evaluation set is only useful if two people grading it independently agree. If your expected output is a paragraph of prose, they will not.

Where you can, make correctness programmatically checkable — the same instinct that makes RLVR work. Does the output contain the right extracted value, parse as valid JSON against the schema, produce the right classification label, pass the test? Those graders run in milliseconds, give the same verdict on the same output every time, and can be used in CI — though, as below, they still need periodic spot-checks against what you actually intend.

Where output is genuinely open-ended, write a short rubric with explicit criteria rather than grading on overall impression, and validate that two people applying it agree before trusting it. A model judge can then apply the same rubric at scale — but check its agreement with human ratings on a slice first, and be aware it will tend to favour longer and more confident answers.

{
  "id": "ext-0041",
  "category": "long_tail",
  "input": "Invoice attached. Total due?",
  "context": "...the document text...",
  "expected": {"total": 1284.50, "currency": "GBP"},
  "grader": "exact_match_json",
  "notes": "Total appears twice; the second is a subtotal. Added after incident 2026-08-14."
}

# category: typical | long_tail | refusal | adversarial
# grader names a function, not a judgement. Graders are ordinary functions,
# versioned with the set:
#   exact_match_json, contains_value, schema_valid, must_refuse, rubric_llm

Keep it uncontaminated

The set is worthless the moment it enters training, and that usually happens by accident — a shared directory, a broad glob, a synthetic generator that saw it — rather than by anyone deciding to.

Hold it out explicitly and separately rather than relying on a random split — and shuffle before splitting, since data that arrived grouped by source will otherwise give you a split that is not representative. Unsloth's troubleshooting notes make this point directly: an unshuffled selection gives you a wrong evaluation.

Store it apart from training data so it cannot be swept into a glob. If you generate training data synthetically, check for near-duplicates of eval items. And if you ever use the eval set to tune hyperparameters, understand that you have partly spent it — which is why a separate validation split for tuning and a test set touched only at the end is worth the extra effort.

Treat it as a versioned asset

The set is code. Put it in version control, review changes to it, and record which version produced which reported number — otherwise an improvement may just be a changed set.

Add to it when you find failures, and resist the urge to remove items the model finds hard. A set pruned of difficult cases reports steadily improving numbers on a steadily easier test, which is the most comfortable way to deceive yourself available in machine learning.

Over a couple of years this becomes the most valuable artefact your team owns. Models, frameworks and base checkpoints will all be replaced. A well-curated evaluation set that encodes what your application actually has to do survives all of it, and makes every replacement decision answerable in an afternoon.

Questions people ask