Open pretraining corpora, with their real sizes and licences

Published 28 September 2026 · 6 min read

Pretraining data is where a model gets almost all of its knowledge. The open corpora differ enormously in how they were filtered and, more awkwardly, in how clearly they are licensed — and the second difference is the one that decides what you can ship.

What pretraining data is, as data

Pretraining consumes raw document text as a single next-token stream. No instructions, no labels, no conversation structure — one text field, loss on every token. That simplicity is why the corpora are measured in trillions of tokens rather than in examples.

It is also the stage that does the heavy lifting. LIMA states the strong version of this as the Superficial Alignment Hypothesis: “a model’s knowledge and capabilities are learnt almost entirely during pretraining”, with later alignment teaching which formats to use. If your model does not know something, more instruction data will not teach it.

Scale, for orientation: 10¹³ tokens and up for a frontier-class run, and 10¹²–10¹³ for a small open model — Llama 2 7B saw 2T tokens, Llama 3.1 8B 15T.

Each stage consumes a different shape of data, usually less of it than the stage before, so the stage your goal belongs to decides what you collect and how much of it.

The corpora, and what they publish

Sizes below are as published by each corpus. Where a licence is marked uncertain, that is because the primary source does not state one — not because it is restrictive. The distinction matters: an unstated licence is a question for your lawyers, not a permission.

Corpus	Published size	Licence as stated
FineWeb	18.5T tokens from 110 Common Crawl snapshots in v1.4 (the paper's v1: 15T from 96)	ODC-By 1.0
FineWeb-Edu	1.3T tokens at quality threshold 3; 5.4T at threshold 2	ODC-By 1.0
DCLM-baseline	3.8T tokens, filtered from a 240T-token pool	CC BY 4.0 (card: intended for research use only)
Dolma	v1.6: 3,059B Llama tokens, 4,367M documents (v1.7: 2.3T)	ODC-By (changed from AI2 ImpACT in April 2024)
Nemotron-CC	6.3T tokens (4.4T real deduplicated + 1.9T synthetic)	not stated in the paper
RedPajama-Data-v2	30T deduplicated tokens, 5 languages	code Apache 2.0; data governed by Common Crawl terms
The Pile	825.18 GiB, 22 components; token count never published	code MIT; no data licence stated
Common Pile v0.1	7.6–8 TB (the paper gives both), 30 sources	built only from openly licensed sources; release licence unspecified
C4 (en)	365M documents, 156B tokens, 305 GB	ODC-BY on the allenai/c4 card; not stated in the documentation paper

A four-panel workflow labelled A to D. A, select a scale, drawn as growing circles, with the options 400M-1x, 1B-1x, 3B-1x, 7B-1x or 7B-2x. B, build a dataset, either by filtering DCLM-Pool (filtering track) or by mixing in external data (mixing track), producing curated data. C, train a language model with a fixed recipe. D, evaluate on 53 downstream zero-shot and few-shot tasks. — DCLM is a benchmark for datasets as much as it is a corpus: at each scale the training code and hyperparameters are fixed, so the only thing that differs between entries is the data. That is what turns claims of the form 'this filter beats that one' into something measured rather than asserted. Image: mlfoundations/dclm · MIT repository · from “DataComp-LM: In search of the next generation of training sets for language models”, arXiv:2406.11794

How FineWeb was filtered, as a worked example

Filtering is where these corpora actually differ, and FineWeb documents its pipeline more fully than most. Text is extracted from Common Crawl WARC files with trafilatura favouring precision, then passed through a URL blocklist, a fastText English classifier at a score threshold of 0.65, and the MassiveText quality and repetition filters.

A line chart of aggregate accuracy (42 to 50%) against training tokens (0 to 350 billion) for base filtering (lowest, about 47%), individual MinHash deduplication, C4 filters and the full FineWeb pipeline (highest, about 49%). — Benchmark accuracy of small models trained on Common Crawl text after each of FineWeb's processing steps: base filtering, per-crawl MinHash deduplication, the C4 filters and FineWeb's own heuristics. Together the steps add about two points by 350 billion training tokens, the largest gain coming from deduplication. No single filter makes the corpus; the stack of them does. Image (cropped): Figure 9 of the paper (left panel) · CC BY 4.0 · from Penedo et al. 2024, “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale”, arXiv:2406.17557

Deduplication is MinHash with 5-gram shingles and 112 hash functions, and — this is the unusual choice — applied per crawl dump independently rather than globally. Three custom filters follow, and the team reports what each cost: requiring that at least 12% of lines end in punctuation removed 10.14% of tokens, a duplicated-line-characters threshold removed 12.47%, and a short-lines filter removed 3.73%. Together about 22% of tokens.

FineWeb-Edu then goes further by scoring for educational quality. A large instruct model rated 460,000 samples from 0 to 5; a smaller embedding model plus a regression head was trained on 410,000 of those ratings, with the remaining 50,000 held out for validation, reaching an F1 of 82% at threshold 3. Applying that threshold removed 92% of FineWeb, leaving 1.3T tokens.

A four-stage pipeline drawn with document icons. Language filtering keeps a Python and a Rust file and drops a JSON file; quality filters apply line-removal rules; content filters mark personal information; decontamination compares documents against an evaluation set and drops one that overlaps with it. — Dolma's chain for code, which is not its chain for web text: filtering by language, line-removal quality rules, removal of personal information, then decontamination against evaluation sets. Every source in a corpus gets its own filters, so 'filtered' in a corpus table means something different for each component. Image: allenai/dolma · Apache-2.0 repository

Aggressive filtering helps, up to a horizon

FineWeb-Edu's filtered 1.3T tokens outperform the full FineWeb on MMLU, ARC and OpenBookQA, and per the paper match the final MMLU of the Matrix corpus with almost ten times fewer tokens. The threshold is a trade-off: per the dataset card, stricter ones significantly degrade HellaSwag and PIQA, which is worth knowing before you adopt it wholesale.

But Nemotron-CC publishes the counter-argument, and it is the most practically useful finding in this whole area. At a 1T-token budget its high-quality subset scores 59.0 MMLU against DCLM's 53.4 — filtering wins. Its full 6.3T-token dataset, trained for the same 1T tokens, scores 53.0, roughly on par with DCLM — but it holds four times as many unique real tokens as DCLM, whose 3.8T is about 80% near-duplicates (some 1T unique). That is the point: aggressive filtering leaves you too few unique tokens to train on for long, so a long run starts repeating.

The practical reading: if you are training for a short horizon, filter hard. If you are training for a long one, you need unique tokens more than you need the highest-scoring ones. Nemotron-CC's answer is to keep more and manufacture more: classifier ensembling and fewer heuristic filters raise the yield of real tokens, and 1.9T of its 6.3T tokens are synthetic rephrasings of real documents, generated with an open 12B instruct model (Mistral NeMo 12B Instruct).

Two panels. Left, 'Empirical Loss (Fixed training length)': final test loss against the fraction of training tokens that are unique, from 100% to 10%, for models of 2.8B, 4.2B and 8.7B parameters trained for 55B, 84B and 178B tokens; loss stays almost flat at first and rises as the unique share falls. Right, 'Predicted Loss (Variable training length)': loss against total training tokens from 10B to 100T, with epoch marks from 1 to 64 and the note 'Repeating for 4 epochs is almost as good as new data'. — At a fixed training length, repeating data costs little at first and more as the unique share falls towards 10% (left); the paper's fit puts four epochs of repetition close to new data, with each further epoch worth less (right). A filter that leaves a long run cycling the same tokens many times has passed that horizon. Image: huggingface/datablations · Apache-2.0 repository · from “Scaling Data-Constrained Language Models”, arXiv:2305.16264

Provenance is a separate question from licence

A corpus can carry a permissive licence and still have uncertain provenance, because the licence covers the packaging rather than every underlying document. RedPajama-Data-v2 is explicit about this: the code is Apache 2.0 while the text itself is governed by Common Crawl's terms of use.

The Pile illustrates the other failure mode. Its code is MIT, no licence is stated for the data, and its Books3 component — roughly 12% of the corpus — became copyright-contested and is no longer distributed, though it remains in the published component table. A corpus is not a fixed object over time.

If you intend to ship a model commercially, the question to ask of a corpus is not whether it is open but whether its provenance is documented well enough to defend. That is a much shorter list.

A horizontal pipeline of six stages applied to web documents, drawn as page icons that are kept or crossed out: language filtering, deduplication by URL, quality filters (a C4 subset plus Gopher rules), content filters for toxic content and PII, deduplication on text overlap, and decontamination against an evaluation set. — The filtering chain between raw crawl and a published corpus. Each stage removes something, so the headline token count tells you almost nothing without this diagram beside it. Image: allenai/dolma · Apache-2.0 repository

Questions people ask

Do I need to pretrain at all?

Almost certainly not. Pretraining from scratch costs millions of GPU-hours and the open base models have absorbed more data than you can afford to. The realistic uses of these corpora are continued pretraining on a domain slice, or sampling from them to build a smaller specialised mix. Starting from a base model is the default for essentially everyone.

What does 'tokens' mean across these numbers?

Not the same thing in every row, which is why cross-corpus comparisons are loose. Dolma counts Llama-tokenizer tokens, FineWeb GPT-2 tokens, C4's documentation counts with a SpaCy tokenizer, and The Pile declines to give a token count at all — reporting bytes plus a tokens-per-byte ratio instead. Treat the figures as order-of-magnitude comparable, not exact.

Is more data always better for pretraining?

For pretraining, scale genuinely does help — unlike instruction tuning, where it often does not. But quality of filtering competes with quantity: in DCLM, MMLU ranged from 35% to 44% at 7B scale purely by changing the filter, with the pool size fixed. Both axes matter, and filtering is the cheaper one to improve.

Can I mix several of these corpora?

Yes, and that is normal practice — Dolma and The Pile are themselves curated mixtures with per-source weights. Deduplicate across sources before mixing, since the web-derived corpora overlap heavily, and keep the per-source weights recorded so the mix is reproducible.

Keep reading

LLM trainingMid-training: the stage between pretraining and fine-tuning LLM trainingDistillation datasets: training on another model's output LLM trainingDPO preference datasets: what to collect and how much