The compute-optimal answer and the right answer are different, and the gap has widened every year since the rule was published. Here are the actual numbers, and why practice left them behind on purpose.
What Chinchilla actually said
Chinchilla found that “the model size and the number of training tokens should be scaled equally” — that “for every doubling of model size the number of training tokens should also be doubled”. The headline conclusion was that “current large language models are significantly undertrained”.
The demonstration: Chinchilla at 70B parameters trained on 1.4 trillion tokens outperformed Gopher at 280B parameters on 300B tokens, at the same compute budget — a quarter the parameters and more than four times the data. Chinchilla reached 67.5% MMLU against Gopher's 60.0%.
One nuance worth getting right, because the figure is repeated everywhere: the paper does not state “20 tokens per parameter” in words. But its Table 3 of compute-optimal model sizes works out to roughly 20 tokens per parameter at every size, from 1B parameters (20.2B tokens) to 1T (21.2T tokens), so the ratio is a fair reading of its scaling law — a rule of thumb, not a quoted rule.
Why practice went past it
Llama 3 is explicit about overtraining on purpose. Its scaling-law extrapolation on its own data “suggests training a 402B parameter model on 16.55T tokens” — roughly 41 tokens per parameter, about twice Chinchilla's implied ratio. And the paper states plainly that “we also train our smaller models for much longer than is compute-optimal”.
In the event it pretrained on about 15 trillion multilingual tokens, against 1.8 trillion for Llama 2 by its count (the Llama 2 paper itself says 2 trillion), with the flagship 405B model seeing 15.6T text tokens.
The reason is inference. A smaller model trained longer costs the same to train as a larger model trained briefly, and far less to serve — every query, forever. If you will run billions of inferences, shifting the budget from parameters to tokens is straightforwardly correct even though it is not compute-optimal for training.
Llama 3's published mix is also worth noting as a template: roughly 50% general knowledge, 25% maths and reasoning, 17% code, 8% multilingual.
| Model | Parameters | Training tokens | Tokens/param |
|---|---|---|---|
| Gopher | 280B | 300B | ~1 |
| Chinchilla | 70B | 1.4T | 20 |
| Llama 2 | 7B–70B | 2T (1.8T per Llama 3) | ~29–286 |
| Llama 3 405B | 405B | 15.6T | ~39 |
| Llama 3 scaling-law estimate | 402B | 16.55T | ~41 |
The real ceiling is unique tokens
Past a point the constraint stops being compute and becomes supply: there is only so much distinct text, and repeating it has diminishing and eventually negative returns.
Nemotron-CC documents this tension precisely. At a 1-trillion-token budget its aggressively filtered high-quality subset scores 59.0 MMLU against DCLM's 53.4 — filtering wins clearly. Trained for the same 1 trillion tokens but drawing on the full 6.3-trillion-token dataset, it scores 53.0, roughly level with DCLM — while holding four times as many unique real tokens as DCLM. Hard filtering wins at a short horizon but leaves too few unique tokens for a long one.
Its answer is to keep more and manufacture more: classifier ensembling and fewer heuristic filters raise the yield of real tokens, and 1.9T of its 6.3T tokens are synthetic rephrasings of real documents. Its 15-trillion-token run reports an 8B model at 70.3 MMLU against Llama 3.1 8B's 65.3.
So the practical rule has two parts. For a short training horizon, filter aggressively — FineWeb-Edu shows the gains. For a long horizon, you need unique tokens more than you need the highest-scoring ones, and the frontier labs are now generating them.
Quality shifts the curve more than count
DCLM is the cleanest demonstration that these ratios are downstream of data quality. Holding the pool fixed and varying only the filter moved MMLU from 35% to 44% at 7B scale.
Its baseline result: a 7B model on 2.6T DCLM-baseline tokens reaching 64% MMLU — state-of-the-art among open-data models, a 6.6 point improvement over the previous best with 40% less compute. And at just 280B tokens it beat Llama 2 7B on MMLU by 5 points with roughly seven times less compute.
Which reframes the question. “How many tokens” has no answer independent of which tokens. A better filter is usually cheaper than more data, and it is the lever most teams have not pulled.
What this means if you are not pretraining
Almost nobody reading this will pretrain from scratch, and the honest advice is not to. The open base models have absorbed more data than you can afford to, and the realistic use of this literature is to inform two other decisions.
Which base model to start from. Tokens seen is a useful signal alongside parameter count: a smaller model trained on far more tokens is often the better starting point, and cheaper to serve.
How to budget continued pretraining. In OLMo 2 the comparable mid-training stage used 50B to 300B tokens — which it frames as 5–10% of training FLOPs — and it is where the filtering lessons above transfer directly, because at that horizon aggressive filtering is the right call.
Questions people ask
Is the 20-tokens-per-parameter rule wrong?
It is not stated in words, but it is what the paper's Table 3 implies at every model size — and it is not what practice does. As a rough guide to compute-optimal training it is a reasonable summary of the paper's scaling fit; as a target it is well below current practice, which runs to 40 tokens per parameter and beyond for models intended to be served at scale.
What happens if I repeat data?
A small number of epochs over high-quality data is normal and fine — phi-1 saw roughly eight passes over about 7B unique tokens. Heavy repetition degrades returns and eventually hurts. This is exactly the constraint that pushes long-horizon runs toward larger pools and synthetic expansion.
Do these laws apply to fine-tuning?
No. Scaling laws here describe pretraining, where you are building capability from scratch. Fine-tuning selects among existing behaviours and is governed by entirely different, much smaller numbers — hundreds to thousands of examples.
How do I count tokens for my own corpus?
With the tokenizer of the model you intend to train, since counts differ substantially between tokenizers. Note that published figures are not always comparable for this reason: Dolma counts Llama-tokenizer tokens, C4's documentation uses a SpaCy tokenizer, and The Pile gives bytes plus a ratio instead of a token count.