The two best-documented SFT recipes sit at a thousand examples and nearly a million, and both report excellent results. That disagreement is not sloppiness — it reflects two different goals, and knowing which one is yours settles how much data you need.
What SFT teaches
Supervised fine-tuning takes a base model that already has the knowledge and teaches it to respond: what format to use, how to follow an instruction, when to refuse. The data is (prompt, response) pairs or multi-turn transcripts — a messages list of roles and content — with the loss computed on the assistant turns only.
Per LIMA, it is explicitly not primarily knowledge acquisition. That framing is what makes the small-data result plausible: if you are selecting among behaviours the model already has rather than installing new ones, you need far fewer examples than intuition suggests.
The two camps, with their numbers
The LIMA position. LIMA fine-tuned a 65B base model on exactly 1,000 carefully curated examples. The result was judged equivalent or preferable to GPT-4 in 43% of cases, 58% against Bard and 65% against DaVinci003. Its ablations are the interesting part: doubling the Stack Exchange training set did not improve response quality, while diversity of source did, and filtered versus unfiltered data made a significant difference. Thirty multi-turn dialogue chains raised excellent responses from 45.2% to 76.1%.
The Tülu position. Tülu 3’s SFT mixture contains 939,344 samples, curated down from a pool of 23,327,961 prompts. The SFT checkpoints average about 60 at 8B and 72.6 at 70B; the final Tülu 3 models, after preference tuning and RLVR on top, reach 65.1 and 76.2. Notably, the paper says they capped the SFT mixture not because more would not help but because “we allocated other prompts for preference optimization”.
Both are true. A thousand examples is enough to teach a style of response. Covering maths, code, safety, instruction-following, multilingual chat and tool use to a uniform standard takes a mixture, because each capability bucket needs its own examples.
The open mixtures and their real sizes
Counts are as published on each dataset's own card or paper. Where a total is absent, it is absent — several of the large NVIDIA mixtures publish category counts but no sum.
| Dataset | Published size | Licence |
|---|---|---|
| Tülu 3 SFT mixture | 939,344 samples, from a 23.3M prompt pool | ODC-BY-1.0; card notes some subsets non-commercial |
| OpenHermes 2.5 | 1,001,551 rows across 15 named sources | no licence stated on the card |
| OpenThoughts3-1.2M | 1.2M rows: 850k maths, 250k code, 100k science | Apache-2.0 |
| Llama-Nemotron-Post-Training | maths 22.1M, code 10.1M, science 709k, IF 56k | predominantly CC-BY-4.0, mixed subsets |
| Nemotron-Post-Training-v2 | chat 628k, STEM 355k, maths 239k, code 175k, ~4.9M multilingual across 5 languages | CC-BY-4.0, mixed subsets |
| SmolTalk | 1.1M pairs; MagPie-Ultra 431k is the largest component | Apache-2.0 for its new subsets; reused sets keep their own licences |
| smol-smoltalk | 460,341 train / 24,229 test | Apache-2.0 |
| LIMA | 1,000 sequences | CC BY-NC-SA, or a stricter source licence (card) |
Which position applies to you
The deciding question is how many distinct capabilities you need the model to have.
One narrow behaviour — summarise these documents this way, classify these tickets, always emit this JSON schema. You are in LIMA territory. Hundreds to a few thousand examples, and your effort is far better spent on consistency and review than on volume. Every example that contradicts the others costs you more than an additional correct one gains.
A general assistant — helpful across maths, code, chat, refusals and several languages. You need a mixture, and you need enough of each bucket. This is where the hundred-thousand-plus counts come from, and where starting from an existing open mixture and adding your own domain slice is far more sensible than building from scratch.
Most people asking the question are in the first case and budgeting for the second.
Composition matters more than count
The useful lesson from the large mixtures is not their size but their structure. Tülu 3 curated nearly a million samples from 23 million candidates — a 4% acceptance rate — and organised them by capability so that coverage was deliberate rather than accidental.
If you build or combine a mixture, record the per-bucket counts and keep the balance deliberate. Merging three public datasets without checking usually means whichever source happened to be largest dominates the model's behaviour, and conflicting answer styles across sources teach it to be inconsistent.
Deduplicate across sources too. The open instruction datasets overlap heavily — several of them list each other as components — and duplicates get silently overweighted.
Questions people ask
What is the smallest SFT dataset worth trying?
Unsloth's documentation suggests a minimum of 100 rows with over 1,000 preferable, and LIMA's result at 1,000 supports that range for a single coherent behaviour. Below 100 you are mostly testing whether the pipeline runs. The honest framing is that a few hundred consistent, reviewed examples is a real experiment, not a toy.
Should I use an existing open mixture or build my own?
Both, in that order. Start from an open mixture with a licence you can live with to get general instruction-following, then add your own domain examples. Building a general mixture from scratch duplicates work that has been done better, and your own data is the part nobody else has.
How many epochs?
One to three, which is both the documented range and consistent with what the large open recipes do. Beyond that you are increasing memorisation rather than capability — watch for training loss dropping below about 0.2, which indicates overfitting, and for validation loss rising while training loss falls.
Does multi-turn data matter if my use case is single-turn?
Less, but it is cheap insurance and LIMA's ablation is striking: adding just 30 dialogue chains raised excellent responses from 45.2% to 76.1%. If users might ever ask a follow-up, a small number of multi-turn examples is worth including, and remember to compute loss on every assistant turn rather than only the last.