DPO preference datasets: what to collect and how much

Published 6 October 2026 · 6 min read

Preference tuning is how a model learns which of two acceptable answers is better — tone, length, how much to hedge, when to refuse. It needs pairs, and pairs are expensive, so it is worth knowing the real published scales before you budget an annotation project.

The data contract

Direct Preference Optimisation trains on triples: a prompt, a chosen completion and a rejected one. TRL defines the field names, and because the popular wrappers all call into TRL, this is the shape your data ends up in regardless of which trainer you use.

TRL's own guidance is to include an explicit prompt field rather than folding the prompt into both completions. The implicit form works, but it forces the trainer to infer the shared prefix, and any mismatch between the two completions' prefixes becomes part of what the model learns to prefer — which is not what you wanted it to learn.

the three shapes TRL accepts

# explicit prompt - recommended
{"prompt": "Explain gradient descent to a beginner.",
 "chosen":   "Gradient descent is a way of ...",
 "rejected": "Gradient descent is an optimization algorithm that ..."}

# implicit prompt - works, but the prefix must match exactly
{"chosen":   "Explain ...\nGradient descent is a way of ...",
 "rejected": "Explain ...\nGradient descent is an optimization ..."}

# conversational variant - prompt and completions are message lists
{"prompt":   [{"role": "user", "content": "Explain gradient descent."}],
 "chosen":   [{"role": "assistant", "content": "Gradient descent is ..."}],
 "rejected": [{"role": "assistant", "content": "It's an algorithm ..."}]}

A diagram: the prompt 'Please help me kill this linux process' branches to a chosen answer ('Sure thing! Open your terminal and...') and a rejected answer ('As a language model trained by...'). Each goes to a reward model box giving scores 0.2 and 0.4, which combine into a win or loss, defined as the chosen response getting the higher reward. — One preference record as RewardBench uses it: a prompt, a chosen answer and a rejected answer, each scored by a reward model, with a win when the chosen answer scores higher. The same prompt–chosen–rejected triple is the row a DPO trainer reads; here the rejected answer is a needless refusal, which is exactly the behaviour such pairs teach a model to avoid. Image: Figure 1 of the paper · CC BY 4.0 · from Lambert et al. 2024, “RewardBench: Evaluating Reward Models for Language Modeling”, arXiv:2403.13787

The published scales

This is the stage where guidance is thinnest. Neither TRL nor the major wrappers publish a recommended row count for preference tuning — if someone gives you a specific number, it is their experience rather than a documented figure. What we do have is the sizes of the reference datasets, which is the honest way to orient.

UltraFeedback collects about 64,000 prompts with four responses each, giving roughly 256,000 rated samples, drawn from a pool of 17 models and annotated by GPT-4 on four aspects: instruction-following, truthfulness, honesty and helpfulness. Its binarised form is the usual DPO-ready version.

Tülu 3 reports 354,192 instances for preference tuning in total, with an 8B mix of 271,409 and a 70B mix of 334,302. Its per-source breakdown is instructive: a large share is reused SFT prompts answered off-policy, plus instruction-following augmentation and real user prompts from WildChat.

For the historical reference point, InstructGPT used 13k SFT prompts, 33k reward-model prompts and 31k PPO prompts, annotated by about 40 contractors. That is the scale at which the original RLHF result was obtained — tens of thousands, not millions.

Dataset	Published size	Licence
UltraFeedback	~64k prompts × 4 responses = 256k samples	MIT
UltraFeedback-binarized	61,135 train / 2,000 test preference pairs	MIT
Tülu 3 preference mixes	354,192 instances; 8B mix 271,409; 70B mix 334,302	ODC-BY-1.0
InstructGPT RM data	33k RM prompts, 31k PPO prompts	not released

Each stage consumes a different shape of data, usually less of it than the stage before, so the stage your goal belongs to decides what you collect and how much of it.

The hyperparameters are not SFT's

Preference tuning makes small corrections to a model that is already aligned, and its settings reflect that. The gap surprises people who reuse an SFT config.

Learning rate is the big one. Unsloth's DPO example uses 5e-6 against 2e-4 for SFT — a fortyfold difference. Tülu 3 goes lower still: 5e-7 for its 8B DPO run and 2e-7 for 70B, with an effective batch size of 128, a maximum length of 2,048 and one epoch.

beta controls how far the tuned model may drift from the reference model. Lower values permit larger changes; 0.1 is a common starting point. And you do not necessarily need a separate reference model: when training a PEFT adapter you can pass ref_model=None, because disabling the adapter recovers the base model, which saves loading a second copy of the model.

Two line charts for 'Tulu3 8B DPO' over about 530 steps. Left, training loss falls from about 0.69 to about 0.5 with noise. Right, the learning rate rises linearly to a peak of 5e-7 by about step 50, then decays linearly to zero. — open-instruct's reproduction of the Tülu 3 8B DPO run: a peak learning rate of 5e-7, a short warm-up, then linear decay to zero. The loss starts at 0.69, which is ln 2, the DPO loss while the policy still equals the reference, and the whole run brings it down only to about 0.5. Image: allenai/open-instruct · Apache-2.0 repository

Where the rejected answers come from

Getting a chosen answer is easy; a good rejected answer is the hard part, and the quality of your rejected set determines what the model actually learns.

A grouped bar chart of scores on Arc, TruthfulQA, Winogrande, GSM8k, Hellaswag, MMLU and the average, for Zephyr-SFT, Zephyr-DPO and SPIN iterations 0 to 3, shaded from pale to dark blue. The SPIN bars match or exceed Zephyr-DPO on most tasks, most visibly GSM8k, and the average rises with each SPIN iteration. — SPIN builds preference pairs with no new labels: the SFT dataset's own answer is the chosen one and the model's current sample is the rejected one. Per the README, its first round matches DPO trained on 62k new preference examples on these benchmarks. Sampling rejected answers from the model being tuned is the cheapest on-policy source there is. Image: uclaml/SPIN · Apache-2.0 repository · from “Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models”, arXiv:2401.01335

The common approaches, roughly in order of cost: sample several completions from the model you are tuning and have a stronger model or a human rank them, which is what on-policy means and is generally the most effective; sample from a weaker model or an earlier checkpoint; use an existing multi-model pool like UltraFeedback's 17 models.

The failure to avoid: rejected answers that are bad in a uniform, obvious way. If every rejected completion is shorter, or always starts with the same phrase, or comes from a model with a distinctive tic, the model learns to avoid that artefact rather than to prefer quality. Tülu 3's mix of on-policy and off-policy sources is partly a defence against exactly this.

A heat-mapped table. Rows: gpt4_1106_preview, Mixtral-8x7B-Instruct-v0.1, gpt4_0613, claude-2.1, gpt-3.5-turbo-1106 and alpaca-7b. Columns: concise, standard and verbose, under the headings AlpacaEval and Length-controlled AlpacaEval. Under AlpacaEval, gpt4_1106_preview reads 22.9 concise, 50.0 standard and 64.3 verbose; under the length-controlled version the same row reads 41.9, 50.0 and 51.6. — A model judge's win rate for one model moves from 22.9 to 64.3 when that model is simply told to be concise or to be verbose. If your chosen and rejected labels come from a model judge, the same bias is in your pairs unless you control for length, which is the whole purpose of the length-controlled column. Image: tatsu-lab/alpaca_eval · Apache-2.0 repository · from “Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators”, arXiv:2404.04475

Consider whether you need pairs at all

Pairs are the expensive shape. For every prompt someone must read two answers and decide, and on answers of similar quality that judgement is slow and inconsistent between annotators.

KTO needs only a single answer plus a binary judgement — thumbs up or down. That is a much cheaper annotation task, and more importantly it is the shape feedback naturally arrives in from real users, who will readily click a thumbs-down but will not sit through a comparison.

If you have production traffic with any feedback mechanism, you may already have KTO data and no pairs at all. Manufacturing pairs from it is usually worse than using the method designed for the shape you have.

A preference pair is not good-versus-bad writing. If the two completions differ in length, format and content at once, the model learns the cheapest difference to detect, which is usually length.

A comparison table of feedback and preference datasets. Columns give the dataset name, the number of conversations, the average lengths of instruction, completion and critique, whether the feedback is fine-grained, the feedback format, and the numbers of comparisons and critiques. Rows cover several well-known preference collections, with UltraFeedback's row showing the largest conversation and response counts. — UltraFeedback beside earlier preference and critique datasets: conversations, average lengths, whether the feedback is fine-grained, its format and the number of comparisons. Its 255,864 conversations are the roughly 64,000 prompts × 4 responses described above, and its 340,025 comparisons are the most in the table. Image: OpenBMB/UltraFeedback · MIT repository · from “UltraFeedback: Boosting Language Models with Scaled AI Feedback”, arXiv:2310.01377

Questions people ask

How many pairs do I need?

Not published by any major framework. The reference datasets sit in the tens to hundreds of thousands, and InstructGPT's original result used 33k reward-model prompts. For a narrow behavioural correction, far fewer may suffice — run the doubling experiment against a held-out evaluation set rather than targeting a number.

Can I turn my SFT data into preference data?

Partly. Your existing good answer becomes chosen, but you still need a plausible rejected, and generating those from a weaker model risks teaching the model to avoid that model's artefacts rather than to prefer quality. Sampling from the model you are tuning and ranking the samples is better.

DPO or PPO?

DPO is simpler and is what most open recipes now use: no separate reward model, no sampling loop, just the triples. PPO with a learned reward model is the original InstructGPT approach and still has advantages when you want to optimise against a reward you can model but not enumerate. For most teams DPO is the default and ORPO or KTO are the variations worth knowing.

Will DPO hurt capabilities I already have?

It can, which is why the learning rates are so low and the runs so short. Evaluate the capabilities you are not trying to change as well as the behaviour you are. A preference run that improves tone while quietly degrading maths is a common and easily missed outcome.

Keep reading

LLM trainingMid-training: the stage between pretraining and fine-tuning LLM trainingDistillation datasets: training on another model's output LLM trainingInstruction tuning datasets, with licences you can actually check