Language models label text well when the categories are clearly defined and the judgement is one an educated generalist could make, and there they are far cheaper than crowd work. They fall short on specialist labels, and their errors are systematic rather than random. Whether a model can replace your annotators is a question that a few hundred human-labelled items answer, and nothing else does.
What the headline studies measured
The result everyone cites is Gilardi, Alizadeh and Kubli (2023), published in PNAS. Across four samples of tweets and news articles, 6,183 items in all, ChatGPT labelled relevance, stance, topic and framing zero-shot. Its accuracy exceeded that of Amazon Mechanical Turk workers by about 25 percentage points on average, and each label cost under $0.003, about thirty times less than MTurk.
Two details in that paper change how it should be read. “Accuracy” means agreement with the trained research assistants, who were the reference, so the study shows ChatGPT came closer to trained annotators than crowd workers did; it cannot show ChatGPT beating them. And ChatGPT's intercoder agreement, 91% at temperature 1 and 97% at 0.2 against 79% for the trained annotators, is agreement between two runs of the same model. That is consistency, which a model can have while being consistently wrong.
Törnberg (2023) used a task with an objective answer: the party of the US politician who wrote each of 500 tweets from the 2020 election. GPT-4 was more accurate and more reliable than both expert coders and crowd workers, ten of whom answered each item. Earlier, Wang et al. (2021) found that reaching the same performance cost 50% to 96% less with GPT-3 labels than with human labels across a range of language understanding and generation tasks, and that combining GPT-3 labels with human ones did better still on a limited budget.
The common thread is the kind of task: short texts, a handful of well-defined categories, and judgements that do not need specialist knowledge. The survey by Tan et al. (2024) covers the wider literature.
| Study | Task | Reference | Finding |
|---|---|---|---|
| Gilardi et al. 2023 | tweets, news: stance, topic, frames | trained assistants | ≈25 pts above MTurk |
| Törnberg 2023 | party of tweet author | actual party | GPT-4 above experts, crowd |
| Ziems et al. 2024 | 25 social-science benchmarks | human labels | below fine-tuned models |
| Pangakis et al. 2023 | 27 tasks, 11 datasets | expert labels | median F1 0.707 |
| Tseng et al. 2025 | finance, law, biomedicine | expert labels | best 47.5–87% |
Where agreement collapses
Pangakis, Wolken and Fasching (2023) replicated 27 annotation tasks from 11 datasets in recently published social-science articles, labelling with GPT-4 and comparing against the original human labels. Median F1 was 0.707, with median precision 0.650 and recall 0.829. Nine of the 27 tasks had precision or recall below 0.5. Two tasks from the same dataset scored F1 0.259 and 0.811, so good performance on one label tells you little about the next.
Ziems et al. (2024) tested 13 models on 25 computational social science benchmarks. On classification the models did not outperform the best fine-tuned models, though they reached fair agreement with humans; on free-form tasks their explanations were often rated above the crowd-written references. That is a useful division: generating text is a strength, assigning a category from a fine-grained scheme less so.
Expertise is where the gap is widest. Tseng et al. (2025) evaluated models on finance, biomedical and legal annotation tasks labelled by experts, 200 items each. The best accuracy per task ranged from 47.5% on assigning legal function categories to 87% on contract clause identification. Chain-of-thought and self-consistency gave marginal or negative gains, and reasoning models did not improve significantly over non-reasoning ones in most settings.
Two further failure modes are worth testing for. Long inputs: Liu et al. (2023) found that models use information at the start and end of a long context better than information in the middle, which matters when the label depends on one sentence of a long document. And subjective labels, such as offensiveness, where people disagree for legitimate reasons: a model returns one perspective with confidence, which settles the disagreement only by ignoring it.
Systematic errors are worse than random ones
Human label noise is partly idiosyncratic. Two annotators tend to err on different items, which is why double labelling and adjudication catch so much of it. A model's errors are correlated: the same prompt and the same weights make the same mistake on every similar item, thousands of times. Running it again does not help, as Gilardi et al.'s 97% run-to-run agreement shows.
Some of the biases are documented. Zhao et al. (2021) showed that few-shot GPT-3 predictions lean towards labels that are frequent in the prompt, labels near the end of it, and tokens common in pretraining data. Reiss (2023) found that small changes to prompt wording, or simply repeating the same input, changed ChatGPT's classifications enough to fall short of usual reliability thresholds, though pooling repeated runs improved matters.
For a training set, this is the dangerous property. A classifier trained on model labels learns the error pattern as if it were signal, and an evaluation set labelled by the same model will not reveal it. Our post on distillation datasets describes the same inheritance when a model's outputs, rather than its labels, are the training data.
Validation: the human-labelled sample you still need
Pangakis et al. propose a workflow that fits most projects: write the codebook, have experts label a random sample, label the same sample with the model using the codebook as the prompt, compare, revise the codebook if needed, then test on human-labelled items held back from the revision. They suggest 250 to 1,250 randomly sampled texts depending on the task and class balance, and more for very rare categories. In their own runs codebook revisions brought only small gains, so the sample is there to measure, not to tune.
The sample size is set by the classes you care about, not by the total. With 400 items, a 95% interval on an overall accuracy near 80% is about ±4 points. But if a class makes up 5% of the data, those 400 items contain about 20 examples of it, and its recall is barely estimated. Stratify so that each important class has enough items of its own.
- Write the guidelines first and use the same text as the prompt; our annotation guidelines template covers the parts that matter.
- Draw a stratified random sample, have it labelled by two people and adjudicated, and record their agreement. Human–human agreement is your ceiling.
- Split the sample into a development part for prompt changes and a held-out part you score once.
- Compare model and adjudicated labels per class: precision, recall and the confusion matrix, plus Cohen’s kappa set beside the human–human kappa.
- Fix acceptance thresholds before you look at the results, including a per-class recall floor.
- Pin the model version, and validate again whenever the model, the prompt or the data source changes.
Hybrid workflows and cost per correct label
The pattern that holds up best is a model first pass with people on the uncertain part. Pangakis et al. ran each text at least seven times and scored how often the runs agreed; items on which every run agreed were 19.4 points more accurate than the rest, and 85.1% of items were in that group. Agreement across repeated runs, or between two unrelated models, is a cheap way to decide which items go to a person.
Route by class as well. If validation shows a class with low precision, every label in it goes to review, whatever the model's agreement says. And keep a small random audit of the auto-accepted labels, because that is the only place the confident errors show up.
Then compare on cost per correct label rather than cost per label. A model label at a tenth of the price of a human one is not a tenth of the cost if a fifth of a class is wrong and nobody knows which fifth: the errors have to be found, and finding them is human work. The useful sum is model cost, plus review of the routed items, plus the audit, divided by the labels that end up correct. The vision equivalent of this workflow, with correction rather than routing, is covered in model-assisted labelling.
Terms of use and the data you send
Model outputs come with terms. OpenAI's terms of use say you may not “use output from the Services to develop models that compete with OpenAI”. The Llama 3.1 community licence requires that if outputs are used to create, train or improve an AI model that is distributed or made available, “Llama” goes at the beginning of that model's name. Labels used to train an internal classifier are a different case from a released general model, but the question is one for whoever reviews your licences, before labelling starts.
Sending text to a hosted model is also processing by a third party. If the text contains personal data, the questions in our GDPR post apply; an open-weight model run on your own hardware avoids the transfer, at the cost of running it.
Finally, record which model, version, prompt and date produced each label. When a provider updates a model behind the same name, labels from before and after are not the same instrument, and you will want to know which is which.
Questions people ask
Can LLMs replace human annotators?
For some labels, yes: short texts, well-defined categories and judgements that need no specialist knowledge, where studies found models matched or beat crowd workers. For expert labels in law, finance or medicine, subjective categories and rare classes, they fall short, and you still need people to build the reference that tells you which case you are in.
Is ChatGPT better than crowd workers at annotation?
On the tasks Gilardi et al. tested, yes: about 25 percentage points more accurate than MTurk workers on average, measured against trained research assistants, at about a thirtieth of the cost. The trained assistants remained the reference, and the result applies to their tasks, tweets and news labelled for stance, topic and framing, not to annotation in general.
How accurate are LLMs at labelling data?
It varies by task more than by model. Across 27 social-science tasks, GPT-4 had a median F1 of 0.707 against human labels, and nine tasks had precision or recall below 0.5. On expert finance, legal and biomedical tasks, the best models scored between 47.5% and 87% accuracy. The number for your task comes only from your own validation sample.
How do you validate LLM-generated labels?
Label a stratified random sample with people, adjudicate it and measure human–human agreement. Run the model on the same items with the guidelines as the prompt, then compare per class with precision, recall and kappa. Pangakis et al. suggest 250 to 1,250 items, more for rare classes. Re-validate whenever the model version or prompt changes.
When should you use humans instead of LLMs for labelling?
When labels need domain expertise, when categories are subjective, when a class is rare and costly to miss, and always for the evaluation set. Use people also for the items a model labels inconsistently across repeated runs, and for a random audit of what it labels with confidence. The model handles volume; people decide what counts as correct.