A strong LLM judge can agree with human raters about as often as they agree with each other on open-ended chat preferences. Outside those conditions agreement varies widely by task, and it drops sharply when the judge cannot answer the question itself. The only way to know where your judge stands is a human-labelled calibration set, measured properly.
What the 80% figure measured
The number comes from Zheng et al. (2023), the MT-Bench and Chatbot Arena paper. Its abstract says strong judges such as GPT-4 achieve “over 80% agreement, the same level of agreement between humans”. The setting behind it is specific: 80 multi-turn questions, answers from six models compared in pairs, and about 3,000 votes from 58 expert labellers, mostly graduate students.
The paper reports two agreement figures. Counting ties and inconsistent judgements as ties, GPT-4 agreed with the human experts 66% of the time, and the experts agreed with each other 63% of the time. Restricted to votes where neither side said tie, the figures were 85% and 81%. The headline uses the second setting.
Agreement also depended on how different the two answers were. The authors report it rising from 70% to nearly 100% as the performance gap between the paired models grew. Telling a strong model from a weak one is easy for any rater; the hard comparisons, between close systems, are where a judge earns its place and where agreement is lowest.
None of this is a flaw in the paper. It is a measurement of one judge on one kind of task, against a human ceiling that was itself well short of 100%. It says little about grading correctness in a specialist domain, or about a different judge model on your data.
The documented biases
Several biases are now well measured. Wang et al. (2024) showed how far position alone can move a verdict: with ChatGPT as the judge, Vicuna-13B won 2.5% of comparisons against ChatGPT when its answer was shown first and 82.5% when it was shown second, and by choosing the order it “could beat ChatGPT on 66 over 80 tested queries”. Zheng et al. found that when the two answers were swapped, GPT-4 gave a consistent verdict in 65.0% of cases, GPT-3.5 in 46.2% and Claude-v1 in 23.8%.
Self-preference is subtler. Zheng et al. saw GPT-4 favour its own answers by about 10 points of win rate and Claude-v1 by 25, while noting that their data could not confirm the bias. Panickssery, Bowman and Feng (2024) then found that models recognise their own text: GPT-4 told its summaries apart from those of other models and humans with 73.5% accuracy, and fine-tuning showed a linear relation between how well a model recognises itself and how strongly it prefers itself.
| Bias | What was measured | Mitigation |
|---|---|---|
| Position | GPT-4 consistent on 65% of swapped pairs (Zheng et al.) | Judge both orders; count a split as a tie |
| Verbosity | A padded “repetitive list” answer fooled Claude-v1 and GPT-3.5 in 91.3% of cases, GPT-4 in 8.7% | Rubric that penalises padding; check score against length |
| Self-preference | Self-recognition correlates with self-preference (Panickssery et al.) | Use a judge from a different model family |
| Leniency | Judges tend to mark answers correct (Thakur et al.) | Track false positives against human labels |
Where judges fail: correctness without a reference
Preference between two fluent answers is the easy case. Grading whether an answer is correct is harder, and the evidence is consistent: a judge is reliable mainly on questions it could answer itself. On ten MT-Bench maths questions, Zheng et al. found GPT-4 misjudged 14 of 20 tests with the default prompt. Asking it to solve the problem before grading brought that to 6, and solving it separately, then passing the solution in as a reference answer, brought it to 3. That works only while the judge’s own solution is right.
Krumdick et al. (2025) tested this directly with 1,200 responses and 3,600 expert correctness annotations, including a new finance question set written by professionals. In pairwise grading against a reference it had written itself, GPT-4o reached a Cohen’s kappa of 0.86 with the experts on questions it had answered correctly, and 0.16 on questions it had got wrong. Given an expert-written reference, its kappa on those same questions rose to 0.83. They also found that 15 of the 40 GPT-4-written reference answers in MT-Bench’s maths and reasoning categories were incorrect or inconsistent.
Breadth tells the same story. Bavaresco et al. (2024) compared 11 LLMs with human judgements across 20 annotated datasets and found reliability varied with the property being judged, the expertise of the human raters and whether the text was written by a person or a model. All models aligned better with humans when judging human-written text, and correlated better with non-expert raters than with experts. Their recommendation is validation and calibration “against task-specific human judgments”.
Measuring agreement properly
Raw percent agreement is the wrong headline statistic. Thakur et al. (2024) evaluated 13 judge models grading answers from 9 models on 400 TriviaQA questions, against three human annotators. The humans agreed with each other on 98.52% of items, a Scott’s π of 96.2. The best judges, Llama-3 70B, Llama-3.1 70B and GPT-4 Turbo, reached π in the high 80s: excellent on any usual scale, and still well below the humans.
Their more useful finding is that percent agreement compressed the judges into a narrow band and hid real differences, while π separated them and tracked how far each judge’s scores drifted from the human scores. Even well-aligned judges assigned scores up to five points away from the human ones, and judges leaned towards marking answers correct.
A sound validation report therefore has four parts: a chance-corrected statistic such as Cohen’s kappa, Scott’s π or Krippendorff’s alpha; the same statistic between humans on the same items; false positives and false negatives separately, since a lenient judge and a harsh one can share an agreement score; and all of it per task category, not only pooled. Our Cohen’s kappa calculator computes kappa with a confidence interval for the two-rater case.
Calibrating the judge, and the format question
Zheng et al. describe three formats. Pairwise comparison asks which of two answers is better; it gave the strongest agreement but, in their words, “may lack scalability” as the number of systems grows, since every pair needs a verdict. Single-answer grading scores each answer alone. It scales, agreed with humans nearly as well when ties were excluded (85% first turn, 84% second), but gave more ties and may miss subtle differences. Reference-guided grading adds a reference answer to the prompt. With an expert-written reference rather than the judge’s own, it is the format to use whenever correctness is at stake.
Calibration closes part of the remaining gap. Wang et al. (2024) combined two fixes: having the judge write its reasoning before scoring and sampling several verdicts, and judging both answer orders. This raised GPT-4’s accuracy against majority human labels from 52.7% to 62.5% (kappa 0.24 to 0.37). Sending the 20% of items where the judge was least consistent to human annotators took it to 73.8% (kappa 0.56), slightly above the average human annotator at 71.7% (0.54).
That last result is the practical design: the judge handles the bulk, its own inconsistency picks out the items that need a person, and humans label those. It is cheaper than human evaluation of everything and more trustworthy than the judge alone.
A human-label budget for a trustworthy judge
How many human labels does validation need? Agreement is a proportion, so the arithmetic is the same as for any error rate: the 95% margin of error on an observed agreement p from n items is about 1.96 × √(p(1 − p)/n). The table solves that for n. It covers one estimate; if you report agreement per category, each category needs its own sample, and the items come from the same kind of golden evaluation set you would build for any evaluation.
Then run validation as a procedure, not a one-off. The judge is only as current as its last check, and a change of judge model, prompt or the system being judged can move every number. For the public benchmarks such judges are often pointed at, see LLM evaluation datasets; the label quality of the human set itself is covered in our pieces on annotation quality control and how many annotators per item.
- Sample items from real traffic, stratified by task category, and freeze them.
- Have two people label each item independently; their agreement is the ceiling. Adjudicate disagreements to produce the reference labels.
- Run the judge with both answer orders and a reference answer wherever correctness is graded.
- Report chance-corrected agreement against the reference labels per category, next to the human–human figure, with false positives and false negatives separately.
- Trust the judge only in categories where it approaches the human ceiling; keep humans on the rest, and route the judge’s inconsistent items to them.
- Re-run the check whenever the judge model, its prompt or the system under test changes.
| Margin wanted (95%) | If agreement is near 90% | Near 80% | Worst case (50%) |
|---|---|---|---|
| ± 10 points | 35 items | 61 items | 96 items |
| ± 5 points | 138 items | 246 items | 384 items |
| ± 3 points | 384 items | 683 items | 1,067 items |
Questions people ask
How accurate is LLM-as-a-judge compared with humans?
On MT-Bench chat preferences, GPT-4 agreed with expert humans 85% of the time on non-tied votes, slightly above the 81% the experts managed with each other; counting ties, 66% against 63%. On other tasks agreement varies widely. On correctness grading it falls sharply when the judge cannot answer the question itself: Krumdick et al. measured a kappa of 0.16 on such questions when the judge had only its own reference answer.
What biases do LLM judges have?
The best documented are position bias (preferring the first or second answer: GPT-4 gave the same verdict after a swap in 65% of cases in Zheng et al.), verbosity bias (preferring longer or padded answers), self-preference (favouring text from the same model, linked to self-recognition by Panickssery et al.) and leniency (marking answers correct too readily). Each can be measured on a human-labelled calibration set.
Do you still need human evaluation if you use an LLM judge?
Yes, in two roles. A human-labelled calibration set tells you whether the judge can be trusted on each task category, and must be re-checked when the judge or the system changes. And a share of items, ideally those where the judge is inconsistent, should keep going to people: Wang et al. found that routing 20% of items to humans brought a calibrated judge level with human annotators.
How many human labels do you need to validate an LLM judge?
For an agreement estimate within plus or minus five points at 95% confidence, 138 items if agreement is near 90%, 246 if near 80%, and up to 384 in the worst case. That is per estimate: reporting agreement for each task category needs a sample per category. Label each item twice so that you also know the human–human ceiling.
Is pairwise or pointwise judging more reliable?
In Zheng et al., pairwise comparison gave slightly higher agreement with humans than single-answer grading when ties were counted (66% against 60%), and the two were level once ties were excluded. Pairwise suffers from position bias and needs a verdict for every pair; pointwise grading scales but gives more ties. For correctness, a reference answer matters more than either format.