The quiet fact about open instruction data is that most of it was written by a model. Knowing which model, and under what terms, matters more than the row count.
What distillation means here
In its classical form, distillation trains a student on a teacher's output distribution — the full logits. In current practice the word almost always means something simpler: generate completions with a strong model, then supervise-fine-tune a smaller model on them.
So the data shape is ordinary SFT data — (prompt, completion) pairs — whose responses happen to be machine-written. Everything about the training is the same; what changes is where the answers came from, and therefore what you are inheriting.
The scale is 105 to 107 examples, which is only affordable because the answers are generated. Paying humans to write ten million responses is not a thing anyone does.
The named teachers and real counts
The better open datasets name their generator, which is the single most useful thing a distillation dataset card can do — it tells you what behaviour you are importing and whose terms apply.
| Dataset | Scale and teacher | Licence |
|---|---|---|
| OpenThoughts3-1.2M | 1.2M rows — 850k maths, 250k code, 100k science; annotated with QwQ-32B | Apache-2.0 |
| Llama-Nemotron-Post-Training | DeepSeek-R1 3,934,627; Qwen-2.5-Math-7B 19,840,970; Qwen-2.5-Coder-32B 8,917,167 | predominantly CC-BY-4.0 |
| Nemotron-CC | 1.9T synthetic pretraining tokens, rephrased by Mistral NeMo 12B instruct | not stated in the paper |
| Llama 3 code SFT | “over 2.7M synthetic examples” for code alone; rejection sampling with K typically 10–30 per prompt | model licence applies |
The licence question is the real constraint
This is where distillation projects fail, and they fail late — after the model is trained.
Two separate permissions are involved. First, the dataset's own licence, which the table above covers. Second, and more easily missed: the terms of the model that generated it. Some model licences explicitly restrict using outputs to train competing models; others are silent; others are permissive. A dataset released under Apache 2.0 whose contents were generated by a model whose terms forbid that use is a problem the dataset licence does not solve.
If you are generating your own distillation data, read the generating model's terms before you start, not after. If you are using someone else's, check what generated it — which is why a card that names its teacher is more trustworthy than one that does not, even though it gives you more to worry about.
What you inherit along with the capability
A student trained on a teacher's outputs acquires the teacher's errors with the teacher's confidence, and this is qualitatively different from training on human data.
Human-written data is wrong in varied, idiosyncratic ways that partly cancel out. A single model is wrong systematically — the same misconception, the same hedging tic, the same formatting habit, across millions of examples. The student learns the pattern, not the noise.
Two practical consequences. Verify where you can: for maths and code the answer is checkable, and filtering generated data by whether it is actually correct is the highest-value step available — which is what rejection sampling does when its acceptance test is a correctness check: generate K candidates and keep the ones that pass. And keep a human-written evaluation set. If both your training and evaluation data come from the same teacher, your benchmark measures agreement with that teacher rather than correctness.
Reasoning traces, specifically
The current wave of distillation targets reasoning: the teacher is a model that produces long chains of working, and the student learns to produce them too.
This works notably well, and the datasets above are mostly of this kind. Two things are worth knowing. The traces are long, so token counts run far ahead of row counts and your sequence length and memory budget need to accommodate them — a million reasoning examples is not comparable in cost to a million short instruction pairs.
And a correct final answer does not mean correct working. Filtering on the answer leaves traces that reach the right result by invalid steps, and the student learns the invalid steps too. Where you can check intermediate steps, it is worth doing.
Questions people ask
Is distilling from a closed model allowed?
It depends entirely on that model's terms of service, which you should read rather than infer. Several explicitly prohibit using outputs to develop competing models. Open-weight models with permissive licences are the safer route for anything you intend to ship, and are what the major open distillation datasets now use — notice that the teachers named above are open models.
How does distillation compare to training on human data?
Cheaper per example by orders of magnitude, and competitive or better on benchmarks for reasoning tasks. Weaker where the teacher is weak, and systematically rather than randomly so. The strongest practical setup is generated data at volume, filtered by verification, with a human-written evaluation set held separate.
Can a student beat its teacher?
On specific benchmarks, sometimes — particularly when the generated data is filtered for correctness, so the student trains only on the teacher's successes rather than its average behaviour. Broadly exceeding the teacher's general capability from its outputs alone is not something to plan around.
Should I say that my model was distilled?
It is good practice and increasingly expected, and it helps anyone evaluating your model understand its failure modes. Naming the teacher and the dataset in your model card costs nothing and is the same transparency you want from the datasets you consume.