RLHF is the technique that made instruction-following models usable, and the data scale behind the original result is smaller than most people assume. Understanding that recipe explains why the field moved on from it.
The original three datasets
InstructGPT is the reference implementation, and it used three distinct datasets at three stages. The paper gives the numbers directly: “the SFT dataset contains about 13k training prompts”, “the RM dataset has 33k training prompts”, and “the PPO dataset has 31k training prompts”. The annotation was done by “a team of about 40 contractors”.
That is the whole thing. Roughly seventy-seven thousand prompts across three stages, and a team you could fit in a room. The result was a 175B model preferred to GPT-3 in 85 ± 3% of comparisons, and 71 ± 4% against few-shot GPT-3.
One frequently repeated claim deserves a correction: the famous line that a 1.3B InstructGPT beat the 175B GPT-3 is stated only qualitatively in the paper. No exact win rate is given for that specific comparison — the figure everyone cites plots win rates against the 175B SFT model instead.
The three stages, and the data each needs
The pipeline's structure is worth knowing even if you use DPO, because the stages map onto different data-collection problems with different costs.
| Stage | Data shape | Who produces it |
|---|---|---|
| 1. Supervised fine-tuning | prompt → demonstration response | humans write ideal answers — the most expensive per example |
| 2. Reward model | prompt + k ranked completions | humans rank model samples — cheaper, no writing |
| 3. PPO optimisation | prompts only | no new annotation; the reward model scores samples |
Why DPO replaced it for most teams
RLHF as originally formulated has three moving parts you must get right: a reward model that generalises, a PPO loop that is stable, and a KL constraint tuned so the policy does not drift. Each is a source of difficulty, and the reward model is a source of a specific one — reward hacking, where the policy finds inputs the reward model scores highly and humans do not.
Direct Preference Optimisation removes the reward model entirely. The same preference triples optimise the policy directly, with no sampling loop and no learned reward to exploit. The data you collect is identical; the machinery is far simpler.
That is why the current open recipes look the way they do. Tülu 3 runs SFT, then DPO, then RLVR, where the RL reward comes from verifiers rather than a learned preference model. The preference data that would once have trained a reward model now trains the policy directly.
When a reward model is still the right choice
Reward models have not disappeared, and there are cases where they remain the better tool.
You want to score many samples cheaply. A reward model is a reusable scorer. Best-of-n sampling at inference, rejection sampling to filter generated training data, ranking candidates in a pipeline — all need a score for an arbitrary completion, which DPO does not give you. Llama 3’s rejection sampling with K typically 10 to 30 candidates per prompt depends on being able to score them.
Online training. If you want the model to learn from its own current samples rather than a fixed dataset, you need something to score those samples as they appear. The online variants of these methods exist for this reason.
A reward you can model but not enumerate. Some objectives are easier to learn from examples than to write down, and sit between the learned-reward and programmatic-verifier cases.
If none of these apply — you have preference pairs and you want a better-behaved model — DPO is simpler and is what the field defaults to.
What to collect, in practice
The useful conclusion is that the data is largely the same whichever method you pick, so collect it in the most flexible form.
Gather prompts that reflect real use, sample multiple completions from the model you intend to improve, and have those completions ranked. From that single collection you can train a reward model, run DPO on the top-versus-bottom pairs, or extract thumbs-up and thumbs-down judgements for KTO. Collecting only binarised pairs keeps all three open but throws away the extra comparisons a full ranking gives.
And sample on-policy where you can — from the model you are actually tuning. Preferences over another model's outputs teach your model about a distribution it does not produce, which is a large part of why Tülu 3's mix deliberately combines on-policy and off-policy sources.
Questions people ask
How many ranked comparisons do I need?
No framework publishes a figure. InstructGPT's reward model used 33k prompts, and Tülu 3's preference mixes run to a few hundred thousand instances. For a narrow behavioural correction, far fewer is plausible — measure against a held-out set rather than aiming at a number.
Is RLHF the same as alignment?
No. RLHF is one technique for shaping model behaviour toward human-judged quality. Alignment is the broader goal, and encompasses data curation, evaluation, red-teaming, safety training and deployment controls. A model can be RLHF-tuned and still badly aligned for a given use.
Can I do RLHF without any human annotators?
You can substitute a strong model as the judge, which is what UltraFeedback does with GPT-4 ratings, and it works well enough to be standard practice. The limits are the judge's own biases and blind spots, and — if you evaluate with the same judge — a benchmark that measures agreement rather than quality. Keep a human-rated evaluation slice.
Does preference tuning make models worse at anything?
It can. Optimising for human-judged helpfulness tends to increase length and hedging, and can trade away capability on tasks the preference data does not cover. Evaluate the things you are not trying to change — a preference run that improves tone while degrading maths is common and easy to miss.