In medical imaging the annotator is the scarcest part of the pipeline, and two equally qualified experts will still draw different boundaries. Medical image annotation therefore starts with deciding whose labels count and how disagreements are resolved, before any tool is chosen.
Who should annotate
The default answer is a specialist for every label, and the default is expensive enough that many projects mix specialists with trained non-specialists. The evidence on where that works is thinner than for natural images, but it exists.
Wahid et al. (Scientific Data, 2023) released a crowdsourced radiotherapy contouring dataset with segmentations from more than 200 annotators across five disease sites, separating experts (board-certified physicians recognised for disease-site expertise) from non-experts, most of whom were practising radiation oncologists. Their usage notes cite work showing that non-expert consensus segmentations could approximate gold-standard expert segmentations. Note what “non-expert” means there: mostly clinicians without recognised expertise in that disease site, not lay crowd workers.
Evaluation sets are where panels are worth their cost. CheXpert’s test set of 500 studies was annotated by eight board-certified radiologists; the majority vote of five formed the ground truth. In the MedSAM annotation study, the two readers were radiologists with eight and six years of clinical practice in abdominal disease.
A defensible split is to have specialists make every diagnostic decision (what the finding is, whether it is present) and to let trained annotators do mechanical work under specialist review, such as refining a boundary a specialist has placed. Whether that holds for your structure is an empirical question, answered by measuring agreement between the two groups on an overlap set.
How much experts disagree, measured
Joskowicz et al. (European Radiology, 2019) had eleven radiologists delineate 3,193 contours of liver tumours, lung tumours, kidneys and brain haematomas on 490 CT slices. For two observers, mean volume overlap variability was 8.8% for kidneys, 17.8% for liver tumours, 18% for brain haematomas and 20.8% for lung tumours. For any two randomly selected observers it ranged from 5% to 57%. Groups of two, three and five observers captured on average 37%, 53% and 72% of the variability, and eight accounted for 75 to 94%. The authors conclude that two or even three observers may not be enough to establish its full range. They also found that 38.5% of the non-agreement came from part of one observer’s contour disagreeing with all the others, which is the case adjudication exists for.
Detection disagrees too. In LIDC-IDRI, each of 1,018 CT cases was read by four experienced thoracic radiologists. Of 7,371 lesions marked as a nodule by at least one radiologist, 2,669 were marked as a nodule of 3 mm or more by at least one; only 928 of those (34.7%) received that mark from all four.
Variability also depends on the structure. Wahid et al. report from related work that target volumes varied more than organs at risk, and that head and neck structures varied more than other disease sites. Well-bounded organs such as the kidney sit at the low end; tumours, whose edges are a clinical judgement, sit at the high end.
For metric choice, Metrics Reloaded (Nature Methods, 2024) asks explicitly how accurate the reference annotation is, and warns about the Dice coefficient on small structures. A one-voxel disagreement is negligible on a liver and a large fraction of a 3 mm nodule, so the metric has to match both the structure and the reference noise. The label type matters as well: in the cost study covered in our comparison of image annotation types, scribbles that worked on natural images underperformed on CT and MRI.
Reference standards, from one reader to STAPLE
Once several readers exist, someone has to decide how their labels become a reference. The options differ in what they cost and in what they hide.
LIDC-IDRI’s protocol is the best-documented multi-reader design. In a blinded phase each radiologist read every scan independently; in an unblinded phase each reviewed their own marks alongside the anonymised marks of the other three and gave a final opinion. The stated goal was to find all nodules “without requiring forced consensus”, so every reader’s marks are kept (see the TCIA collection page). Users choose their own threshold, such as agreement by three of four.
STAPLE (Warfield, Zou and Wells, 2004) estimates a hidden true segmentation and each rater’s sensitivity and specificity by expectation-maximisation. Wahid et al. built separate expert and non-expert consensus masks with SimpleITK’s STAPLE at a threshold of 0.95.
| Reference | How it is built | What it hides |
|---|---|---|
| Single reader | one specialist per case | the reader’s style and all variability |
| Independent reads kept | every reader’s label stored, as in LIDC | nothing; users must choose |
| Majority vote | k readers, majority wins (CheXpert test) | minority opinions; ties |
| Adjudication | a senior reader resolves disagreements | how much disagreement there was |
| STAPLE | EM estimate of truth and rater performance | bias shared by all raters |
| Pathology or outcome | biopsy, surgery or follow-up decides | boundaries: it settles what, not where |
Labels mined from reports
The cheapest labels in radiology are already written, in reports. CheXpert labelled 224,316 chest radiographs of 65,240 patients with a rule-based labeller in three stages, mention extraction, classification and aggregation, giving each finding a value of positive, negative, uncertain or not mentioned. Its macro-averaged F1 was 0.948 for mention extraction, 0.899 for negation and 0.770 for uncertainty.
The uncertain label is the interesting part. “Cannot exclude pneumonia” is information about the image and the reader, not noise, and CheXpert compared five ways of training on it: ignoring uncertain labels, mapping them to negative, mapping them to positive, self-training, and treating uncertainty as its own class. The best policy differed by finding: mapping uncertain to positive did best for atelectasis, a separate class did best for cardiomegaly, and for consolidation, oedema and pleural effusion the best and worst policies were not significantly different.
Report-mined labels scale; they do not replace image reading for evaluation. CheXpert itself used radiologist panels reading the images for its validation and test sets. Do the same: mine reports for training volume, and have specialists read the test images directly.
DICOM, NIfTI and the tools
Clinical images arrive as DICOM, which carries patient identifiers in its headers and sometimes burnt into pixels; our post on de-identification and its limits covers what removing them involves. Research pipelines often convert to NIfTI, a single volume with an affine that maps voxels to world coordinates. Segmentations go back out as label-map volumes, DICOM SEG objects or RTSTRUCT contour sets, and each conversion between voxel masks and contours re-rasterises the boundary.
Coordinates are where silent errors live. 3D Slicer’s coordinate-system documentation states that DICOM and most medical imaging software use LPS (left, posterior, superior), while Slicer uses RAS internally; the two differ only in the sign of the first two axes, and Slicer may flip them when it reads or writes files. A mask produced in one convention and read in the other loads without error, mirrored. Check every new tool chain on a case with an asymmetric, known structure before a batch.
Model-assisted tools now do much of the drawing. In the MedSAM study (Nature Communications, 2024), two radiologists annotated adrenal tumours in 10 CT cases, 733 tumour slices, with and without the model, and annotation time fell by 82.37% and 82.95% respectively. MedSAM was trained on 1,570,263 image-mask pairs across 10 modalities. MONAI Label brings interactive, model-assisted labelling into 3D Slicer and the web-based OHIF viewer. Assistance speeds the drawing; it does not settle any of the reference-standard questions above. For data to start from, see our list of open medical imaging datasets.
Specifying a medical annotation order
A medical annotation order needs more than a class list. These are the items to settle in writing:
- Readers: qualification (board certification, sub-specialty, years of practice) and the number of readers per case, which can differ between training and test sets.
- Reading protocol: independent blinded reads, any unblinded second phase, and who adjudicates.
- Reference standard: how reads are combined (majority, STAPLE, adjudication, pathology) and a commitment to deliver the raw reads as well.
- Structure definitions: anatomical boundaries, inclusion rules, and the display settings (window and level) under which they are drawn; our annotation guidelines template gives the general structure.
- Size floor: the smallest finding to mark and how smaller ones are recorded, as LIDC did with its under-3 mm category.
- Format and coordinates: DICOM SEG, RTSTRUCT or NIfTI, the coordinate convention, and a check case.
- Agreement: overlap cases read by every reader, with Dice and a surface distance for contours and a chance-corrected statistic for categorical labels; our Cohen’s kappa calculator handles the two-reader case.
- Data handling: de-identification before images leave the source, and the access terms.
Questions people ask
How are DICOM and NIfTI images annotated for machine learning?
In a viewer that reads the volume with its geometry, such as 3D Slicer, often with model assistance from MedSAM or MONAI Label. The annotator draws slice by slice or in 3D, and the result is saved as a NIfTI label map, a DICOM SEG object or an RTSTRUCT contour set. The main technical risk is coordinates: DICOM uses LPS and some tools use RAS, so check orientation on a known case.
Who should annotate medical images?
Specialists should make the diagnostic decisions, and test sets should be read by several of them, as CheXpert did with eight radiologists and a majority of five. Trained non-specialists can do mechanical work under review. Wahid et al. cite evidence that non-expert consensus can approximate expert contours, but their non-experts were mostly practising clinicians. Measure agreement on an overlap set before relying on any split.
How do you create ground truth for medical image segmentation?
Have several qualified readers segment independently, keep every read, then combine them by a stated rule: majority vote, adjudication by a senior reader, or STAPLE, which estimates a consensus and each reader’s sensitivity and specificity. Pathology or follow-up can confirm what a finding is but not where its boundary lies. Report the inter-reader agreement alongside any model result.
How much do radiologists disagree on segmentations?
Measurably, and by an amount that depends on the structure. In Joskowicz et al., eleven radiologists contouring CT structures showed mean two-observer volume overlap variability from 8.8% for kidneys to 20.8% for lung tumours, and 5% to 57% for random pairs. In LIDC-IDRI, only 34.7% of lesions marked as nodules of 3 mm or more by at least one of four radiologists were marked so by all four.
What tools are used for medical image annotation?
3D Slicer is a widely used open-source viewer and editor, and MONAI Label adds model-assisted labelling to it. MedSAM, a segmentation model trained on over 1.5 million medical image-mask pairs, cut annotation time by about 82% for two radiologists in its study. Whatever the tool, check that it preserves DICOM geometry and exports in the format your pipeline expects.