RLVR is the stage that needs the least data collection and gives the clearest signal — when your task happens to be programmatically checkable. The catch is in that condition, and it is worth being precise about where it holds.
What it is
Tülu 3, which named the technique, describes it as “an RL-based method that only gets a reward if the model’s completions are verified to be correct” — it “trains the model on verifiable rewards instead of a reward model”.
That substitution is the whole idea. Classic RLHF trains a reward model to predict human preference, then optimises against it, which means inheriting the reward model's errors and its exploitability. If correctness can instead be checked by a program — does the arithmetic come out right, do the unit tests pass, does the output satisfy the stated constraint — you can use the checker directly and skip the learned reward entirely.
The data shape is (prompt, ground-truth answer) or (prompt, verifier program). No preferred and rejected pairs, no human ratings, and in the GRPO formulation no target output at all — just prompts plus a function that scores the model's own samples.
Tülu 3's prompt mix, exactly
Tülu 3 publishes its verifiable prompt dataset in full, and the scale is the striking part — under thirty thousand prompts against an SFT mixture of over nine hundred thousand.
| Source | Prompts | Verifier |
|---|---|---|
| GSM8K train | 7,473 | exact match against the extracted answer |
| MATH train | 7,500 | exact match against the extracted answer |
| IF verifiable | 14,973 | prompt-specific verifiers |
| Total | 29,946 |
Where verifiability actually holds
The method is only as good as your checker, so it is worth sorting tasks honestly.
Cleanly verifiable — arithmetic and symbolic maths with an extractable final answer; code against a test suite; structured output against a schema; format and length constraints; extraction tasks where the ground truth is a record you hold; translation into a formal language that either parses or does not.
Partly verifiable — tasks where a program can check necessary conditions but not sufficiency. Code that passes the tests may still be unmaintainable; a citation may exist but not support the claim. Useful, with the understanding that you are optimising the checkable part.
Not verifiable — tone, helpfulness, tastefulness, how much to hedge, whether an explanation suits a beginner. These need preference data, and no amount of cleverness turns them into a program.
The honest risk is specification gaming: the model optimises exactly what you measure. If your verifier accepts an answer for the wrong reason, the model will find that route, and it will do so more reliably than a human tester would.
GRPO, and why prompts are all you need
The trainer most people reach for here is GRPO, and TRL classifies its dataset type as prompt-only — no completions. Any extra column, such as a reference answer, is passed through to your reward function as a keyword argument.
The loop: for each prompt the model generates several candidate answers, your reward function scores each one, and the relative scores within the group drive the update. No human writes a target output at any point, which is why this stage can be so much cheaper than the ones around it.
What it costs instead is engineering. The reward function has to be fast enough to call thousands of times, sandboxed if it executes model-generated code, and precise about partial credit. Writing a good verifier is real work — but it is work you do once, against data collection you would otherwise do continuously.
# the dataset: prompts plus a reference answer for the reward to read (no completions)
{"prompt": "What is 17 * 23? End with '#### <answer>'.", "ground_truth": "391"}
# the reward function does the verifying
import re
def reward_exact_match(completions, ground_truth, **kwargs):
scores = []
for c, gt in zip(completions, ground_truth):
m = re.search(r"####\s*(-?[\d.]+)", c)
if m is None:
scores.append(-0.5) # no parseable answer at all
elif m.group(1).strip() == str(gt).strip():
scores.append(1.0)
else:
scores.append(0.0)
return scores
# several candidates per prompt are sampled and scored relative to each other,
# so the signal comes from the spread within the group
What it delivered
Tülu 3 reports its 8B model at an average of 65.1 and its 70B at 76.2, with 8B IFEval 82.4, GSM8K 87.6 and MATH 43.7, and 70B IFEval 83.2, GSM8K 93.5 and MATH 63.0. Its 405B RLVR run reports an average of 80.7 on the same basis (80.0 excluding safety).
The thing to notice is which benchmarks moved: instruction-following, grade-school maths and competition maths — precisely the three capabilities its 29,946 verifiable prompts target. RLVR is a focused instrument, and the published results are consistent with that rather than with general improvement.
The released mixes are available as allenai/RLVR-GSM-MATH-IF-Mixed-Constraints with single-domain splits alongside. Check the licence on each card individually — one of them ships a Llama licence file, which suggests a derived-licence constraint rather than a straightforwardly open release.
Questions people ask
How many prompts do I need for RLVR?
Tülu 3's recipe uses 29,946 across three domains — one of the few precisely published figures in this area. That is the number to anchor on, with the caveat that it was tuned for three specific capabilities. A single narrow capability plausibly needs far fewer, and the cost of finding out is low since prompts are cheap.
Can I combine RLVR with DPO?
Yes — Tülu 3's pipeline runs SFT, then preference tuning, then RLVR as its final stage. They address different things: preference tuning handles what cannot be checked, RLVR sharpens what can. In Tülu 3's recipe RLVR comes last, on top of SFT and DPO. RL directly on a base model also works for reasoning — DeepSeek-R1-Zero and Open-Reasoner-Zero do exactly that, and it is the “Open R1-Zero” step in Hugging Face's Open R1 plan — but DeepSeek reports poor readability and language mixing from R1-Zero, which its full R1 recipe addresses with supervised data.
What stops the model gaming the verifier?
Nothing automatic, and this is the real risk. Make the verifier strict about form as well as content, include negative rewards for unparseable output, and inspect samples during training rather than only watching the reward curve — a reward climbing while quality does not is the signature of gaming.
Is RLVR the same as GRPO?
No. RLVR is the idea — reward by program rather than by learned model. GRPO is one algorithm for optimising against such a reward, and the more common choice in open reasoning recipes. Tülu 3 itself ran its RLVR stage with PPO and a programmatic reward.