How much label noise a model tolerates depends less on the error rate than on its shape. Random flips spread evenly across classes do surprisingly little harm to a large training set; the structured errors real annotators make do much more. Errors in the test set are worse again, because they corrupt the number you use to choose between models.
Three kinds of noise
A noisy label is one that differs from the label the guidelines would give. The literature, summarised in the survey by Song et al. (2022), sorts noise by what the probability of a wrong label depends on. The distinction matters because many robustness results use the first kind, while real human noise, where it has been measured, looks like the third.
Instance-dependent noise is the hardest to simulate, so papers rarely do. Wei et al. (2024), in a scoping review of noisy labels in medical prediction, describe instance-based injection as “arguably the most compelling simulation for label noise” and note that it is seldom applied in practice.
| Type | Wrong-label probability depends on | Typical source | Example |
|---|---|---|---|
| Symmetric (uniform) | Nothing: any wrong class equally likely | Simulations; random clicks | A cat labelled as any of nine other classes at random |
| Class-conditional (asymmetric) | The true class | Confusable classes; a guideline gap | Trucks labelled as automobiles more often than as birds |
| Instance-dependent | The item itself | Blur, occlusion, ambiguity, fatigue | A distant, half-hidden truck labelled as an automobile |
The robustness results, read carefully
Two well-known results frame the question. Zhang et al. (2017) showed that standard convolutional networks “easily fit a random labeling of the training data”, and that explicit regularisation does not change this qualitatively. A network will fit whatever labels it is given; the question is what it learns on the way.
Rolnick et al. (2017) then reported test accuracy above 90% on MNIST “even after each clean training example has been diluted with 100 randomly-labeled examples”, with similar behaviour on CIFAR and ImageNet and across several noise patterns, including noise biased towards confusing classes. The title, “Deep Learning is Robust to Massive Label Noise”, is quoted far more often than the conditions.
Three conditions matter. The noisy examples were added to the clean ones rather than replacing them, so every correct label was still in the data. The authors state that training in this regime needs a significant increase in dataset size, related to the dilution factor. And under uniform noise the correct label stays the most common one. With 100 random labels per clean example spread over ten digit classes, about ten land on each class, so each digit carries its correct label about eleven times for every ten of each wrong one. The signal survives because the errors are spread evenly, and that is the property real annotation errors lack.
The practical reading: a large training set with diffuse, random errors of a few percent is not an emergency. A small training set with errors concentrated on particular classes is a different situation, and the robustness papers do not speak to it. Our piece on data quality versus quantity covers the wider evidence on cleaning versus collecting more.
What real human noise looks like
CIFAR-N is the cleanest measurement of human label noise on a standard benchmark. Wei et al. (2022) had 747 Mechanical Turk workers relabel the 50,000 CIFAR-10 training images, three workers per image, and collected one label per image for CIFAR-100.
The single-annotator labels on CIFAR-10 disagreed with the original labels 17.23% to 18.12% of the time, depending on which of the three label sets is used. The majority of three cut that to 9.03%; the worst of three, 40.21%. On CIFAR-100, a single annotator was wrong 40.20% of the time on the 100 fine classes and 25.60% on the 20 superclasses.
The errors were not uniform. They concentrated in pairs of similar classes, such as truck and automobile in CIFAR-10, the animal classes among themselves, and pairs like snake and worm or fox and wolf in CIFAR-100. Hypothesis tests rejected the assumption that the noise was independent of image features, so the class-conditional model usually assumed was too simple as well.
The structure costs accuracy. On the worst-of-three CIFAR-10 labels, a model trained with standard cross-entropy reached 77.69% test accuracy. With synthetic class-dependent noise built to match, it reached 86.69%. Nine points of the damage came from which items were mislabelled, not how many. Robust training methods narrowed the loss considerably: DivideMix reached 92.56% on the same human labels.
Medicine shows a similar range. The review by Wei et al. (2024) cites inter- and intra-observer variability of 5% to 40%, depending on the specialty and dataset. For the effect of adding annotators to bring such rates down, see how many annotators per item.
Networks learn the pattern first and the noise later
Arpit et al. (2017) found that while deep networks can memorise noise, they “tend to prioritize learning simple patterns first”, and that tuned regularisation such as dropout can slow the memorisation of noise without hurting generalisation on real data. Training is, roughly, two phases: the network fits the consistent part of the labels, then starts fitting the exceptions, including the wrong ones.
Several practical techniques exploit this. Early stopping on a clean validation set halts training before the second phase dominates. Sample-selection methods, one of the families Song et al. (2022) survey, treat examples with persistently high loss as probably mislabelled and down-weight or drop them. The same per-example loss makes a cheap review queue: the examples the model still gets wrong late in training are a sensible place to start relabelling.
The caveat comes from real labels. Wei et al. (2022) report that networks “memorize features more easily when learning with real-world human annotations than synthetic ones”. A plausible reason is that human errors fall on items that genuinely resemble the wrong class, so they look like signal. The gap between the two phases is narrower on real labels, and a loss-based filter tuned on synthetic benchmarks should be expected to catch less than it promises.
Noise in the test set is the expensive problem
Training noise costs some accuracy. Test noise corrupts the measurement used to decide what to ship. Northcutt, Athalye and Mueller (2021) estimated an average of at least 3.3% label errors across the test sets of ten widely used datasets, and at least 6% in the ImageNet validation set. About half of the candidates their algorithm flagged, 51% on average, were confirmed as errors by human reviewers.
The errors change which model looks best. On ImageNet with corrected labels, ResNet-18 outperforms ResNet-50 if the share of originally mislabelled test examples rises by just 6%; on CIFAR-10, VGG-11 outperforms VGG-19 at 5%. A benchmark with a few percent of errors can rank models in the wrong order, and the larger model is the one it tends to flatter.
Hekler et al. (2020) show the same effect in a medical setting. They labelled 804 biopsy-verified melanoma and naevus images two ways: by the majority decision of dermatologists and by the biopsy result. When the training and test labels came from the same source, accuracy was 75.03% and 73.80%. When they came from different sources, it fell to 64.53% and 64.24%. The same model, scored against different reference labels, lost about ten points.
So the test set is where label quality is bought first: full redundancy, expert adjudication of disagreements, and an acceptance check, as described in our guides to building a golden evaluation set and to annotation quality control.
Fixing the model or fixing the labels
There are two ways to respond to noise. Training-side methods leave the labels alone and make learning tolerant of them: robust loss functions, reweighting, sample selection and label correction, catalogued by Song et al. (2022). The medical review found robust losses the most popular approach, with reweighting and curriculum learning the most widely used changes to the training procedure, and recommends starting with those easy-to-implement options.
Relabelling fixes the data. It costs annotation time, and it is the only option for a test set, since no training trick repairs a measurement. For training data, the choice depends on how much noise there is, where it sits and how large the dataset is. A workable rule:
- Always relabel the test and validation sets, with two independent labels per item and an expert decision on every disagreement.
- Relabel training data when the noise is structured. If a double-labelled sample shows errors concentrated in a few class pairs, fix the definition first, then relabel those classes. Structured noise is where synthetic-noise results are least reassuring.
- Relabel when the noisy classes are the ones that matter to the product, whatever the overall rate.
- Relabel when the dataset is small. The tolerance in the robustness papers was bought with much more data than most teams have.
- Use a training-side method instead when noise is diffuse, a few percent, and the dataset is large; a robust loss or early stopping costs less than a relabelling pass.
Questions people ask
How much label noise can deep learning tolerate?
With uniform random noise and a large dataset, a lot: Rolnick et al. kept MNIST test accuracy above 90% with 100 random labels per clean example. Those conditions rarely hold. Real human noise is structured, and on CIFAR-10N a standard model lost about nine more points on human errors than on synthetic ones built to match. For small datasets or concentrated errors, a few percent can matter.
What is the difference between symmetric and asymmetric label noise?
Symmetric, or uniform, noise flips a label to any other class with equal probability, so errors spread thinly and the correct label stays the most common one. Asymmetric, or class-conditional, noise flips labels mostly towards particular classes, such as truck to automobile. It concentrates errors on one wrong answer, which a model can learn as if it were a rule, so it does more damage at the same rate.
Do neural networks memorise noisy labels?
Yes, eventually. Zhang et al. showed networks can fit completely random labels. Arpit et al. found that they learn simple, consistent patterns first and memorise noise later, which is why early stopping and loss-based sample selection work. The CIFAR-N authors found real human errors are memorised more easily than synthetic ones, so the margin these techniques rely on is smaller in practice.
How do noisy labels affect model performance?
In training data they lower accuracy, by an amount that depends on the noise structure and dataset size more than the raw rate. In test data they distort the measurement: Northcutt et al. showed that a few percent more test errors can reverse which of two models looks better. That makes test-set noise the more expensive problem, because it misleads every decision made on it.
What is a typical label error rate in human-annotated data?
Curated benchmark test sets average at least 3.3% errors, according to Northcutt et al., with at least 6% on the ImageNet validation set. Single crowd annotators on CIFAR-10N were wrong about 17% to 18% of the time, and a majority of three about 9%. In medical data, reported observer variability ranges from 5% to 40% by specialty.