This argument is usually conducted on intuition. It does not need to be: there are several well-documented results at different scales, and they agree on a nuanced answer rather than a slogan.
At instruction-tuning scale, quality wins decisively
LIMA is the sharpest result. A 65B base model fine-tuned on exactly 1,000 curated examples was judged equivalent or preferable to GPT-4 in 43% of comparisons, 58% against Bard and 65% against DaVinci003.
The ablations matter more than the headline. Doubling the Stack Exchange training set did not improve response quality. Using diverse sources did improve it significantly. Filtered versus unfiltered data from the same source produced a measurable difference. And adding just 30 multi-turn dialogue chains raised excellent responses from 45.2% to 76.1%.
The mechanism is that every instruction example is a demonstration of behaviour. Adding a good one helps; adding a bad one actively teaches something you do not want, and the good one does not cancel it. That asymmetry is why curation dominates at this stage.
At pretraining scale, the filter is a bigger lever than the pool
DCLM ran the controlled version of this experiment: hold the data pool fixed, vary only the filter. At 7B scale, MMLU across filter choices ranged from 35% to 44%.
Nine points of MMLU from the filter alone, with the same underlying data. Its resulting baseline — 3.8T tokens selected from a 240T-token pool — trained a 7B model to 64% MMLU, a 6.6 point improvement on the previous best open-data model with 40% less compute. At only 280B tokens it beat Llama 2 7B on MMLU by five points with roughly seven times less compute.
FineWeb-Edu is the same lesson by a different route. Scoring documents for educational quality and keeping only the top band removed 92% of FineWeb, leaving 1.3T tokens — and that remainder outperforms the full corpus on MMLU, ARC and OpenBookQA, matching the final MMLU of the Matrix corpus with almost ten times fewer tokens, per the paper. Worth knowing: the paper picked that threshold as a trade-off against HellaSwag, and the dataset card says stricter thresholds significantly degrade HellaSwag and PIQA, so it is not free.
Deliberately written data beats scraped data
phi-1 pushed the quality argument furthest. A 1.3B-parameter model, slightly over 50 billion total tokens seen, trained on about 6B tokens of filtered code plus under 1B tokens of generated Python textbooks and around 180M tokens of generated exercises.
It reached 50.6% pass@1 on HumanEval and 55.5% on MBPP, trained in four days on eight A100s plus a further seven hours. Models orders of magnitude larger, trained on far more scraped code, did worse.
Two caveats to carry with that result. The paper releases the model but does not state that the training data was released, and omits the synthetic-generation details for proprietary reasons — so it is a finding rather than a recipe. And it reports the exercises component's size inconsistently across the paper.
The counter-result: filtering can go too far
Nemotron-CC is the finding that keeps this from being a slogan, and the one most worth knowing.
At a 1-trillion-token budget, its high-quality subset scores 59.0 MMLU against DCLM's 53.4 — aggressive filtering wins clearly. Trained for the same 1T tokens, its full 6.3-trillion-token dataset scores 53.0, roughly level with DCLM, but it contains four times as many unique real tokens as DCLM, which is what a long run needs.
The point is that heavy filtering leaves too few unique tokens to sustain a long run, so you begin repeating, and repetition has diminishing returns. For a long-horizon run you need unique tokens more than you need the highest-scoring ones.
Its solution was to filter less and generate more: it ensembled quality classifiers, stopped applying heuristic filters to high-quality documents, and added synthetic rephrasings, which make up 1.9T of its 6.3T tokens. The 15-trillion-token run reports an 8B model at 70.3 MMLU against Llama 3.1 8B's 65.3.
The synthesis
Reading all five together gives a rule with a condition rather than a slogan.
- Instruction tuning: quality dominates, almost without qualification. A thousand curated examples beats a hundred thousand scraped ones. Spend on review.
- Short-horizon pretraining: filter aggressively. The filter is a larger lever than the pool size, worth up to nine MMLU points on identical data.
- Long-horizon pretraining: you need unique tokens. Over-filtering becomes the bottleneck; generate or source more rather than tightening further.
- Any stage: a bad example costs more than a good one gains, which is why deduplication and removing outright broken data are the cheapest wins available.
Questions people ask
What counts as a quality filter in practice?
At pretraining scale: language identification, deduplication, heuristic document filters (punctuation, line length, repetition), and increasingly a learned classifier scoring documents for usefulness — FineWeb-Edu's educational-quality scorer is the clearest published example. At instruction scale: deduplication, format validation, and human review.
Is deduplication part of quality?
It is the highest-value single step at every scale. Duplicates get overweighted, waste compute and inflate memorisation. Every major corpus does it — FineWeb uses MinHash per crawl dump, RedPajama-v2 flags exact duplicates with a Bloom filter, which cut its head and middle partitions by roughly 40%, and ships MinHash signatures so you can add fuzzy deduplication yourself.
Should I remove low-quality data or fix it?
Removing is cheaper and usually right. Fixing is worth it when the data is scarce and expensive — a domain where you have only a few thousand real examples. Nemotron-CC takes a third route at scale: rewriting low-quality documents with a model rather than discarding them.
How do I know whether my data is the problem?
Run the doubling experiment. If the evaluation metric improves when you double the dataset, quantity was binding. If it does not move, quantity is not your constraint and you should look at quality, task definition or the base model.