The answer depends on two things: how accurate one annotator is, and whether annotators make their mistakes independently. When they do, three labels and a majority vote remove most errors and a fifth label buys little. When they do not, extra votes repeat the same mistake. Either way, the cheapest route to the result of three labels is to buy two and add a third only where they disagree.
One, three or five labels: the arithmetic
Start with the idealised case: a yes-or-no label, annotators who are each right with probability p, and errors that are independent of one another. The majority of three is right whenever at least two are right, and the same logic extends to five or seven. The table is that calculation, nothing more.
Two patterns are visible. The first extra labels do most of the work: at 90% single-annotator accuracy, going from one label to three removes 72 of every 100 errors, and going from three to five removes most of the rest at the same cost again. And redundancy helps least where accuracy is lowest: at 70%, five labels still leave about one item in six wrong.
The independence assumption is the weak point. Real errors cluster on hard items, where most annotators make the same wrong call. Take a batch in which 80% of items are easy (each annotator 97% accurate) and 20% are hard (60% accurate). One label gives 89.6% overall; three give 92.7%; five, 93.6%; seven, 94.2%. The hard items gain slowly, from 60% to 71% with seven labels, because voting cannot correct an error most voters share. That is a guideline problem, and our piece on annotation quality control covers how to find it.
| Single-annotator accuracy | 1 label | Majority of 3 | Majority of 5 |
|---|---|---|---|
| 70% | 70.0% | 78.4% | 83.7% |
| 80% | 80.0% | 89.6% | 94.2% |
| 90% | 90.0% | 97.2% | 99.1% |
| 95% | 95.0% | 99.3% | 99.9% |
What Snow et al. and Snapshot Serengeti measured
Snow et al. (2008) ran five language tasks on Amazon Mechanical Turk, from emotion ratings to textual entailment and word-sense disambiguation, and compared pooled non-expert labels with expert annotators. For emotion ratings, pooled across seven labels, matching one expert’s agreement took only four non-expert annotations per item. The figure varied by emotion: two for anger, disgust and sadness, seven for joy, nine for surprise, and fear never reached expert level. On textual entailment, majority voting over ten workers peaked at 89.7% accuracy.
The cost side explains why the paper became a reference. The study collected 21,690 labels for $25.82 at 2008 Mechanical Turk prices. Those prices are not a guide to paying a trained workforce today, but the ratio still holds: several cheap labels can stand in for one expensive one on tasks where non-experts are individually reasonable.
Swanson et al. (2016) give the same picture at scale. Snapshot Serengeti volunteers, more than 28,000 of them, classified 1.51 million camera-trap images, each seen by about 27 people on average. The plurality answer agreed with experts on 97.9% of 3,829 verified images. Bootstrapping smaller crowds, the authors found 90% species accuracy with five classifications and 95% with ten, approaching 98% only after twenty. Rare species had more false positives and false negatives, and images on which volunteers split were much less likely to be right.
Get another label, or label another item?
Redundancy competes with coverage for the same budget. Every second label on an existing item is a first label not bought for a new one. Sheng, Provost and Ipeirotis (2008) studied this trade-off directly and reported four results: repeated labelling can improve label quality and model quality, but not always; when labels are noisy it can beat single labelling even when labels are not especially cheap; once preparing an unlabelled item has a cost, labelling everything several times can pay; and repeated labelling of a carefully chosen set of items is generally better than repeating everything.
A real-world measurement of what repeated labelling buys comes from CIFAR-N. Wei et al. (2022) collected three Mechanical Turk labels for each of the 50,000 CIFAR-10 training images. A single annotator’s labels were wrong 17% to 18% of the time; the majority of three, 9.03%. Three labels roughly halved the noise, at three times the labelling cost.
Whether halving the noise is worth tripling the cost depends on what limits the model. If the learning curve is still steep, new items usually buy more. If the model has plateaued and its errors track the label errors, or if the labels are going into a test set, redundancy wins. Our piece on label noise looks at how much noise a model tolerates and which kind does the damage.
Majority vote and its blind spots
Majority vote is the default because it needs nothing but the labels. Its weaknesses all follow from treating every vote the same.
- Equal weights. A careful annotator and a careless one count the same. On a small, well-trained team this matters less; on a crowd with a long tail of low-effort workers it matters a great deal.
- Ties. Two annotators who disagree, or three who give three different answers in a multi-class task, produce no majority, and the tie-break rule then decides the label.
- Shared errors. If most annotators misread the same definition, the majority is confidently wrong, and a high agreement figure hides it.
- No estimate of anyone’s reliability. It gives a label but tells you nothing about which annotators to retrain, which a model-based method does as a by-product.
Dawid–Skene and its successors
Dawid and Skene (1979) proposed the model that most later methods extend. Each annotator gets a confusion matrix: the probability that they answer j when the truth is i. The true labels and the matrices are estimated together with the EM algorithm, without any gold data. An annotator who reliably confuses two classes is down-weighted on exactly those classes, and an annotator who answers at random ends up with a matrix that carries no information.
Later models add what Dawid–Skene leaves out. GLAD (Whitehill et al., 2009) models item difficulty alongside annotator expertise. MACE (Hovy et al., 2013) gives each annotator a probability of spamming, learnt without supervision, and reports considerable improvements over standard baselines in both label accuracy and trustworthiness estimates. Snow et al. (2008) used a simpler variant with gold items, weighting each worker’s vote by its log-likelihood ratio, and gained 4.0 accuracy points on textual entailment and 3.4 on event ordering over majority vote, averaged across crowd sizes from two to ten.
How much this buys depends on the data. In the comparison by Paun et al. (2018), Dawid–Skene reached 0.92 accuracy against 0.90 for majority vote on the entailment data, and 0.94 against 0.88 (macro F1 0.87 against 0.73) on a sparser crowd dataset, Phrase Detectives, where it was the best model tested. Their guidance: “It is rarely appropriate in crowdsourcing to assume that all annotators have the same ability.” The models need each annotator to have labelled enough items, overlapping with others, to estimate their matrix; with a handful of items per person, they overfit.
The methods are a few lines of code. Crowd-Kit, an Apache-2.0 library, implements majority vote, Dawid–Skene, GLAD, MACE and several others behind one interface.
import pandas as pd
from crowdkit.aggregation import MajorityVote, DawidSkene
df = pd.read_csv("labels.csv") # columns: task, worker, label
mv = MajorityVote().fit_predict(df)
ds = DawidSkene(n_iter=100).fit_predict(df)
# items where the two methods disagree are worth a look
disputed = (mv != ds).sum()Adaptive redundancy and what it costs
The cheapest improvement on fixed redundancy is to stop collecting labels once the answer is settled. For binary labels, the simplest scheme is two plus one: take two labels, and buy a third only when they disagree. Its result is identical to the majority of three, because a third label cannot overturn two that agree, and it needs a third label only on the disagreeing share of items.
Snapshot Serengeti ran a richer version of the same idea. An image was retired when its first five classifications all said it was empty, after ten “nothing here” answers, after ten matching species classifications, or after 25 species classifications in all, so easy images used few labels and contested ones used many. Swanson et al. (2016) found that evenness, a measure of how evenly volunteers split, was the single best predictor of whether the consensus answer was correct. Disagreement is the signal to spend more.
The table costs the options for binary labels at 90% single-annotator accuracy, with independent errors, per 1,000 items. The last column is the marginal price of each error removed, in labels.
| Scheme | Labels per 1,000 items | Errors left | Labels per error removed |
|---|---|---|---|
| 1 label | 1,000 | 100 | – |
| 2 + 1 on disagreement | 2,180 | 28 | 16 (vs 1 label) |
| Majority of 3 | 3,000 | 28 | 28 (vs 1 label) |
| Majority of 5 | 5,000 | 9 | 103 (vs 3 labels) |
Questions people ask
How many annotators do you need per item?
For training data from a trained team, one label per item with a double-labelled sample for quality control is usually enough. Use three, or two plus a third on disagreement, when single-annotator accuracy measured on gold items is below what the data needs, when labels go into a test set, or when a class is costly to get wrong. Beyond five, gains are small unless annotators are individually weak.
Is majority voting a good way to aggregate labels?
It is a reasonable baseline and hard to beat when annotators are skilled and similar. It treats every vote equally, so it suffers when quality varies across annotators or some answer at random. In Paun et al.’s comparison, Dawid–Skene beat majority vote by 2 and 6 accuracy points on two crowd datasets, and every method tied on the other two.
What is consensus labelling?
Consensus labelling collects labels for the same item from several annotators working independently and combines them into one answer, by majority vote or by a model such as Dawid–Skene that weights annotators by estimated reliability. The level of agreement doubles as a quality signal: items where annotators split are the ones most likely to be wrong and worth sending to an expert.
When is it better to relabel than to label new data?
Sheng, Provost and Ipeirotis found that repeated labelling pays when labels are noisy and grows more attractive as unlabelled items become costly to obtain, and that relabelling a selected subset beats relabelling everything. In practice: relabel when the model has plateaued and its errors follow label errors, or for any test set; add new items while the learning curve is still rising.
How many non-expert labels equal one expert label?
In Snow et al.’s 2008 study, pooled emotion ratings needed four non-expert labels per item to match one expert, with a range from two to nine across emotions, and one emotion never matched. There is no general exchange rate: it depends on how hard the task is for non-experts, and for some labels, like fear in that study, no number of crowd votes reached expert level.