How much data do you need for a LoRA?

Published 7 September 2026 · 6 min read

The honest answer has two parts: there are a few published starting points, and a lot of widely repeated numbers that are rules of thumb. Knowing which is which saves you from over-collecting and from under-collecting.

What is actually published

Unsloth's documentation recommends "a bare minimum of at least 100 rows of data" for fine-tuning, with "over 1,000 rows" preferable, and 1–3 epochs. For synthetic generation it suggests having at least 10 real examples to seed from. Its reinforcement-learning guide suggests at least 500 rows for GRPO, and its LoRA hyperparameters guide gives starting points for rank and alpha, covered below.

For row counts, that is close to the whole of the primary vendor documentation. TRL publishes dataset formats and no volumes. Neither Unsloth nor TRL publishes row-count guidance for DPO, ORPO or KTO.

So when you read that DPO needs some specific number of pairs, that figure is someone's experience rather than a documented recommendation. It may well be good advice. It is not a specification, and the distinction matters when you are budgeting an annotation project.

Why 1,000 good examples beats 100,000 scraped ones

Instruction tuning does not teach a model new capabilities so much as select among behaviours it already has. That is why the data requirement is so much smaller than for pretraining, and why quality dominates volume so decisively.

Every example is a demonstration. A thousand examples that all demonstrate the behaviour you want give a clean, consistent signal. A hundred thousand scraped examples containing contradictory answer styles, truncated outputs, wrong formats and the occasional refusal teach the model to sometimes do each of those things — and you cannot undo a bad demonstration by adding a good one.

The practical implication for a data budget: spend it on review rather than on volume. Having a person read and fix a thousand examples is usually a better investment than collecting ten thousand more.

Two line charts titled 'GPT-3 175B validation accuracy vs number of trainable parameters', one for WikiSQL and one for MNLI-matched, plotting Fine-Tune, PrefixEmbed, PrefixLayer and LoRA as separate marker series against a log10 trainable-parameter x-axis. — LoRA's accuracy against trainable-parameter count on GPT-3 175B. The LoRA line stays roughly flat across a wide range of sizes while the prefix-tuning baselines fall away: small ranks already capture most of the gain, which is why rank is rarely the first thing to change. Image: microsoft/LoRA · MIT repository

Rank, alpha and what they do and do not control

LoRA's rank sets how much capacity the adapter has. It is natural to assume a bigger dataset needs a bigger rank, and in a loose sense that is true, but the relationship is much weaker than people expect — ranks in the 16 to 32 range cover a wide span of dataset sizes well for typical instruction tuning, though on harder targets such as code and maths higher ranks measurably help. Unsloth's own examples use 16, and its hyperparameters guide lists 8, 16, 32, 64 and 128 and advises choosing 16 or 32.

Two line charts. A: accuracy on HumanEval against training tokens from 0.25 to 20 billion, with full fine-tuning rising to about 0.26 and LoRA ranks 16, 64 and 256 lower at about 0.16, 0.20 and 0.22. B: accuracy against epochs 1 to 16, with full fine-tuning and LoRA rank 256 near 0.42 to 0.50 and rank 16 lowest near 0.32 to 0.36. — HumanEval accuracy for Llama-2-7B trained on code with LoRA at ranks 16, 64 and 256 and with full fine-tuning: continued pretraining on up to 20 billion tokens (A) and instruction fine-tuning over 16 epochs (B). Higher rank closes some of the gap, but full fine-tuning stays ahead in most settings. Rank caps how much the model can change, whatever the dataset size. Image (top row, cropped): Figure 1 of the paper · CC BY 4.0 · from Biderman et al. 2024, “LoRA Learns Less and Forgets Less”, arXiv:2405.09673

Alpha is a scaling factor applied to the adapter's contribution. Setting it equal to the rank, or to twice the rank, are the two starting points the same guide recommends. These are conventions rather than derived values.

The honest framing: rank is not where your run will go wrong. If results are poor, the dataset is a far more likely cause than the rank, and raising the rank on a small dataset mostly buys faster overfitting.

During training a LoRA adapter never materialises ΔW: it stores the two thin factors and applies them as two small matrix multiplies. For deployment you keep them separate or merge BA into W once. Rank sets capacity, so a rank too low for the shift you are asking for cannot be fixed with more data or more epochs.

Reading the loss curve for too-little-data

Your loss curve will tell you about dataset size if you know what to look for, and one of the thresholds is published.

Unsloth's LoRA guidance states that a training loss below 0.2 means the model is likely overfitting. As a rule of thumb, a loss that settles somewhere around 0.5 to 1.0 is usually healthy, though where it should settle depends on the dataset and task.

A loss that plunges towards zero within the first epoch on a small dataset is the signature of memorisation: the model has learned your specific examples rather than the behaviour they demonstrate. More epochs will not help. More varied data will. Conversely a loss that barely moves suggests the task is not learnable from this data — wrong format, wrong masking, or a mismatch between what you are asking and what the examples show.

Loss exactly 0.0 from the start — not overfitting but a masking bug: every label is -100. Check your response-only masking against the model's template.
Loss dives below 0.2 quickly — overfitting. Fewer epochs, more data, or more varied data.
Loss settles around 0.5–1.0 — generally healthy, task depending.
Loss flat and high — learning rate too low, wrong data shape, or the examples do not actually demonstrate the task.
Training loss falls, validation loss rises — the clearest overfitting signal, and the reason to hold out a validation split at all.

A line chart titled 'loss' with two smoothed series, lora_finetune and qlora_finetune, over about 1,000 steps, above a faint unsmoothed trace. Both fall from about 1.7 to about 0.9 within the first 200 steps, then flatten at roughly 0.85. — LoRA and QLoRA runs of Llama 2 7B from torchtune's QLoRA tutorial. Both drop quickly, then settle around 0.85 and stay there, the healthy shape described above, and the 4-bit base barely changes the curve. A run on a small dataset that kept sliding towards 0.2 would be the memorisation signature instead. Image: pytorch/torchtune · BSD-3-Clause repository

A sizing procedure

Rather than guessing a target up front, let the curve tell you where you are.

Line chart titled "Training and Validation Loss per Epoch", epochs 0 to 5. The blue training loss falls from about 8.9 to about 4.4 at epoch 1, 2.6 at epoch 2 and about 0.4 by epoch 5; the orange validation loss is about 3.8 at epoch 1, dips to about 3.6 at epoch 2, then rises to about 4.5 at epoch 4 and about 5.5 at epoch 5. — Gemma 3 270M fully fine-tuned for five epochs on 20 short player-and-alien exchanges, with 5 held out. Training loss falls to about 0.4 while validation loss is lowest around epoch 2 and then climbs to about 5.5: the widening gap only a held-out split can show. The cookbook accepts it here, because memorising a game persona is the goal. Image: google-gemini/gemma-cookbook · Apache-2.0 repository

Start with whatever you have, even a few hundred examples, and a short run of one to two epochs. Hold out a validation split and shuffle before splitting — Unsloth's troubleshooting notes make the point that an unshuffled selection gives you a wrong evaluation, which is an easy and silent mistake when your data arrived grouped by source.

Then double the dataset and run again. If the validation metric improves, data was your bottleneck and it is worth collecting more. If it does not move, more of the same data will not help: the limit is quality, task definition, or the base model. That single doubling experiment is worth more than any published row count, because it measures your problem rather than someone else's.

Train on a quarter, a half and all of your data, then read the shape. It answers “how much data do I need” for your task rather than in general.

Questions people ask

How many examples for a simple format-following task?

These are the cheapest things to teach, and the low hundreds are often enough — consistent with the documented 100-row minimum. Teaching a model to always emit valid JSON in a fixed schema is largely a matter of showing it the schema repeatedly with no counter-examples. Consistency matters far more than count here: a handful of examples with the wrong shape will undo a great many correct ones.

Does QLoRA need more data than LoRA?

No. Quantisation affects memory and numerical precision, not how many examples the adapter needs to learn a behaviour. The QLoRA paper reports 4-bit QLoRA preserving full 16-bit fine-tuning performance, and a 2024 benchmark run reported no accuracy degradation from Unsloth's QLoRA path relative to standard QLoRA. Choose 4-bit or 16-bit on the basis of the VRAM you have.

Should I train for more epochs if I have little data?

Up to a point, and the documented range is 1–3. Beyond that you are mostly increasing memorisation — the loss keeps falling while generalisation does not improve. If three epochs on your data is not enough, the answer is more varied data rather than more passes over the same examples.

How much data for continued pretraining rather than instruction tuning?

A different order of magnitude entirely — millions to billions of tokens rather than hundreds of examples, because you are teaching knowledge rather than behaviour. Note also that continued pretraining needs the embedding layers in the target modules and a separate, smaller embedding learning rate; Unsloth's docs use 5e-5 with an embedding rate of 5e-6.

Keep reading

Fine-tuningtorchtune has wound down: what to use instead Fine-tuningFine-tuning with Unsloth: what your dataset has to look like Fine-tuningLLaMA-Factory datasets: the dataset_info.json mapping layer