If you only read one format specification, read TRL's. Unsloth trains with TRL's trainers directly, and the other popular wrappers hand their preference training to TRL, so for DPO and its relatives, whatever shape your data starts in, this is the contract it ends up satisfying.
Format versus type
TRL draws a distinction worth adopting. Format is whether your data is standard — plain strings — or conversational, a list of role-tagged messages. Type is what the fields mean: language modelling, prompt-completion, preference, unpaired preference, or prompt-only.
Both axes are independent. Preference data can be standard or conversational. The trainer cares about the type; the chat template cares about the format. Keeping the two ideas separate makes the documentation much easier to follow.
# language modeling
{"text": "The sky is blue."}
# prompt-completion
{"prompt": "The sky is", "completion": " blue."}
# preference, explicit prompt (recommended)
{"prompt": "The sky is", "chosen": " blue.", "rejected": " green."}
# preference, implicit prompt
{"chosen": "The sky is blue.", "rejected": "The sky is green."}
# unpaired preference (KTO)
{"prompt": "The sky is", "completion": " blue.", "label": True}
# prompt-only (GRPO, RLOO, online DPO) - no target at all
{"prompt": "The sky is"}Which trainer wants which type
This table is the reason to read TRL's docs before picking a method. The data you can realistically collect often determines the method rather than the other way round.
| Trainer | Dataset type | What you must collect |
|---|---|---|
SFTTrainer | language modeling or prompt-completion | one good answer per prompt |
DPOTrainer | preference, explicit prompt preferred | two answers plus an ordering |
RewardTrainer | preference, implicit prompt preferred | two answers plus an ordering |
KTOTrainer | unpaired preference, or preference | one answer plus a thumbs up or down |
GRPOTrainer | prompt-only | prompts plus a reward function |
RLOOTrainer; online DPO (experimental) | prompt-only | prompts plus a reward signal |
| ORPO, CPO (experimental) | preference | two answers plus an ordering |
KTO is the cheap one, and that is the point
Paired preference data is expensive. For every prompt an annotator must read two answers and decide which is better — and on answers of similar quality that judgement is slow and inconsistent between people.
KTO needs only a binary judgement on a single answer. That is a fundamentally cheaper annotation task, and it is also the shape feedback naturally arrives in from real users, who will click a thumbs-down on a bad answer but will not sit through a comparison.
If you have production traffic with any feedback mechanism, you may already have KTO data and no preference pairs at all. That is a reason to consider the method, not a reason to manufacture pairs from it.
GRPO needs no answers whatsoever
The prompt-only types are worth understanding even if you do not use them, because they change what counts as a dataset.
For GRPO you supply prompts and a reward function. The model generates several candidate answers per prompt, the function scores them, and the relative scores drive the update. No human ever writes a target output.
This only works where correctness is programmatically checkable: does the code pass the tests, does the arithmetic come out right, does the output parse as valid JSON against a schema, does the extracted value match the record. Where that holds it is extremely attractive, because prompts are cheap and the reward function is written once. Where it does not — anything requiring taste, tone or open-ended judgement — you are back to collecting answers.
Conversational data in any of these types
Every type above has a conversational variant where the string fields become message lists. A conversational preference example has a prompt that is a list of messages, and chosen and rejected that are each a list containing the assistant's reply.
TRL applies the model's chat template for you when it detects the conversational format, which is the behaviour you want — it means the same dataset trains correctly against different model families without you rendering template tokens into your files.
Questions people ask
Does TRL publish how many preference pairs I need?
No. TRL documents the formats and not the volumes. Unsloth publishes row counts for SFT (at least 100, over 1,000 preferable) and for GRPO (at least 500), but none for preference pairs. For preference tuning, look at the scale of the established public preference datasets for orientation and treat any specific recommendation as a rule of thumb.
Can I convert SFT data into preference data?
Partly. Your existing good answer becomes chosen; you still need a plausible rejected. Generating it from a weaker model or an earlier checkpoint is a common approach, but be careful — if the rejected answers are bad in an obvious, uniform way, the model learns to avoid that artefact rather than to prefer quality.
What is beta in the DPO config?
It controls how far the tuned model is allowed to move from the reference model. Lower values permit bigger changes, higher values keep it close to where it started. Unsloth's DPO example uses 0.1. It interacts with the learning rate, and preference tuning generally wants a much lower rate than SFT — around 5e-6 rather than 2e-4.
Do I need a reference model for DPO?
The implementation needs one, but you can pass ref_model=None when training a PEFT adapter: disabling the adapter recovers the base model, so it serves as its own reference. That is what Unsloth's DPO notebook does, and it avoids loading a second copy of the model.