This is the most-asked question in machine learning and the one most badly served by lists. The right dataset is a function of four things, and once you have answered them the field narrows from thousands of options to a handful.
First: are you teaching knowledge or behaviour?
These need different data and different training stages, and conflating them is the most expensive mistake in the sequence.
Knowledge — the model does not know your domain's vocabulary, entities or conventions. Medical abbreviations, your product catalogue, a low-resource language, the statutes of one jurisdiction. This wants continued pretraining on raw domain text: unlabelled documents, a single text field, loss on every token.
Behaviour — the model knows the domain but will not respond the way you need. It is too verbose, will not produce valid JSON, ignores your format, answers when it should refuse. This wants supervised fine-tuning on input-output pairs that demonstrate the behaviour, with loss on the outputs only.
If the model is confidently wrong about facts, that is knowledge. If it knows the answer but presents it badly, that is behaviour. Most people who think they need the first actually need the second, which is fortunate: behaviour needs far less data.
Second: how much data does that stage actually need?
The orders of magnitude differ by a factor of millions across stages, which is why 'how much data do I need' has no single answer.
The numbers below are order-of-magnitude orientation, not thresholds. The only vendor-published figures in the list are the SFT row, where Unsloth's documentation recommends a minimum of 100 examples with over 1,000 preferable, and the GRPO row, where its reinforcement-learning guide suggests at least 500. For preference tuning no major framework publishes a row count, so treat any specific number you are given there as a rule of thumb. The figure after the table shows the typical size of public datasets at each stage, not minimums.
| What you want | Stage | Rough scale | Data shape |
|---|---|---|---|
| Domain vocabulary and facts | Continued pretraining | 10M–10B+ tokens | raw text |
| A task done reliably | Supervised fine-tuning | 102–106 examples | prompt → completion |
| A style or judgement preference | DPO / ORPO | not published; look at reference sets | prompt + chosen + rejected |
| Cheaper signal than pairs | KTO | not published | prompt + completion + thumbs up/down |
| Verifiable correctness | GRPO / RLVR | 500+ prompts (Unsloth, for GRPO) | prompts + a reward function |
| Knowing whether any of it worked | Evaluation | a few hundred, hand-checked | prompt + expected |
Third: what are you allowed to ship?
This is the filter that disqualifies the most promising candidates, and the one people apply last instead of first. It is worth applying before you download anything.
Many widely cited datasets are research-only. Several large image sets are distributed as URL lists whose underlying images carry their own separate rights. Some permissive licences are share-alike, which has implications for what you publish downstream. Some are perfectly open but were assembled from sources whose own licensing was never established — open in licence, uncertain in provenance.
The question is not 'is this dataset open' but 'can a model trained on this be used the way we intend, by the people we intend, in the jurisdictions we operate in'. If the answer is research-only and you are shipping a product, the dataset is not a candidate however good it is. Decide this first and you stop wasting evaluation effort on data you cannot use.
Fourth: does it match your inference distribution?
A dataset that looks ideal on paper can still be the wrong choice because it does not resemble what your model will actually see.
Check the obvious axes. Language and register — formal written prose will not prepare a model for messy user chat. Length — a model trained on short answers produces short answers. Image resolution, sensor and lighting. Class balance against your real traffic. Age, if the domain has drifted since collection.
The practical test: sample twenty rows at random and read them as if they were production inputs. If they feel like a different problem from yours, no amount of volume fixes that. Twenty rows read carefully tells you more than any summary statistic on the dataset card.
A procedure you can follow
Putting it together, in the order that wastes the least effort:
- Write down the behaviour you want as a sentence, then decide: knowledge, behaviour, or preference. That picks your stage.
- Filter all candidates by licence against your actual deployment, before downloading.
- Read twenty random rows of each survivor. Discard anything that does not resemble your inference distribution.
- Build the evaluation set before the training set — a few hundred examples you have checked by hand, held out and never trained on. Without this you cannot tell whether any dataset choice helped.
- Start with the smallest plausible training set and a short run. A thousand good examples that move your eval beats a hundred thousand that you never measured.
- Only scale the thing that moved the number. If more data did not help, the bottleneck is data quality, task definition or the base model — not volume.
When the answer is that no dataset fits
Sometimes you work through this honestly and nothing qualifies. The task is specific to your domain, the open sets are the wrong distribution, and the one that would work is research-only.
That is a real outcome, not a failure of searching, and it has two answers. Generate synthetic data from a stronger model where the task permits it and the licence of that model's output allows your use. Or commission the data — which for a few thousand well-specified examples is often faster and cheaper than people assume, and is the one route that gives you data nobody else has.
Both are better than training on a dataset you already know is the wrong shape and hoping volume compensates.
Questions people ask
Can I just combine every dataset I can find?
You can, and for continued pretraining mixing broadly is normal. For instruction tuning it tends to disappoint: conflicting answer styles teach the model to be inconsistent, and duplicates across sources get overweighted. If you combine, deduplicate first, standardise the format, and check that the mixture's balance resembles your traffic rather than whichever source happened to be largest.
Is a bigger dataset always better?
Past a surprisingly low threshold, no. For instruction tuning, several thousand carefully curated examples routinely beat hundreds of thousands of scraped ones, because every low-quality example is a demonstration of behaviour you do not want. For pretraining, scale genuinely does help — the stages behave differently, which is why the distinction in the first section matters.
Should I use synthetic data?
It works well for teaching format and structure, and for augmenting a small real set. It works poorly for teaching facts the generating model does not reliably know, since you inherit its errors with its confidence. Check the generating model's terms for whether its outputs may train another model, and always keep a real, human-checked evaluation set.
How do I know whether the dataset or the base model is my problem?
Prompt the base model carefully on your eval set first, with few-shot examples. If good prompting gets most of the way there, your problem is behaviour and a modest SFT set will finish the job. If the base model cannot do it even with help and examples, you are missing knowledge or capability, and more instruction data will not supply either.