Model-assisted labelling: what SAM and Grounding DINO actually save

Three ways a model can help

“Model-assisted labelling” covers three different workflows, and a speed-up measured on one does not transfer to the others.

Pre-label and correct. A model labels the batch first and annotators fix the boxes, masks or tags. The saving depends on how many pre-labels are right, because a badly wrong mask can take longer to repair than to draw.

Interactive. The annotator places a click or a box, a promptable model returns a mask, and the annotator refines it with more clicks. SAM and SAM 2 were built and measured this way.

Verify only. The annotator accepts or rejects machine labels without editing them. It is the cheapest per item and the most exposed to the bias described below, because rejecting takes effort and accepting takes none.

Text has the same three patterns. Letting a language model label text with no correction at all is a fourth case with its own evidence, covered in LLMs as annotators.

The measured speed-ups

Most published claims about auto-labelling come from tool vendors and cannot be checked. The exceptions are the data engines Meta described in its Segment Anything papers, which report human time per mask or per frame as the model improved.

In the Segment Anything assisted-manual stage, annotators clicked and the model proposed masks. Average time per mask fell from 34 to 14 seconds as the model was retrained between rounds. The authors put 14 seconds at 6.5 times faster than COCO mask annotation (COCO's own 22 worker-hours per 1,000 masks gives about 5.6 times) and about twice as slow as a bounding box drawn with extreme points. That stage produced 4.3M masks from 120k images.

The next stage is the instructive one. In the semi-automatic stage the model filled in the masks it was confident about and people annotated what was left. Average time went back up to 34 seconds per mask, because the remaining objects were the hard ones. Pre-labelling removes the easy work first, and what is left costs what it always did.

SAM 2 measured the same thing for video. Running SAM on every frame took 37.8 seconds per frame; with SAM 2 propagating masks between frames it took 4.5 seconds, 8.4 times faster, and annotators edited about 19% of frames rather than all of them. A separate group of annotators still checked every masklet.

SAM 3 automates the checking as well. Llama models fine-tuned on human accept and reject decisions judge whether a mask is good and whether every instance was found. The paper reports that this roughly doubles throughput against a human-only pipeline, and notes that in new domains the exhaustivity check needs modest domain-specific human supervision.

Correction time is the real metric

A pre-label saves time only if correcting it is faster than starting from nothing, and that depends on how good it is. The cleanest evidence is old and comes from text. Fort and Sagot (2010) pre-annotated Penn Treebank sentences with part-of-speech taggers of different accuracy. With the best tagger, one annotator went from about 7.5 minutes per ten sentences to about 2.5 and the other from about 11.5 to 2.5. Only the weakest tagger, at 66.5% accuracy, slowed annotation significantly.

Lingren et al. (2014) measured dictionary pre-annotation of clinical named entities in 1,400 clinical trial announcements. Time savings were 13.85% to 21.5% per entity, and inter-annotator agreement held at 93.4% to 95.5%. That is a real saving and a modest one: each span still had to be read and judged.

Masks follow the same arithmetic. A user study on agricultural images, described in our instance segmentation post, timed hand-drawn polygons at a median of about 9 seconds per mask against roughly 1.3 to 3.2 seconds with SAM-based tools. If your annotators spend most of their time on a few difficult masks per image, a median saving will overstate what you see.

So log time per accepted label rather than time per item opened, and split it by what happened to the pre-label: kept, edited, deleted or redrawn. A tool that reports only total hours cannot tell you which of those is driving the cost.

Anchoring: pre-labels change what annotators see

The less visible cost is that people correcting machine output are pulled towards it. In the same part-of-speech study, an annotator working from the best pre-annotation made more systematic errors of one kind, tagging adjectives where the reference had past participles, and both annotators said they concentrated more without pre-annotation. With a weaker tagger, one annotator's accuracy fell from 98.1% to 95.8%. The authors concluded that the quality gain is real and comes with biases that annotators should be told about.

Not every study finds it. Lingren et al. found no significant change in agreement or annotator performance with dictionary pre-annotation. The difference may lie in the errors: a dictionary fails visibly by missing terms, while a fluent model is wrong in ways that look right.

A randomised experiment by Beck, Eckman, Kern and Kreuter (2025) tested this directly with 2,784 crowd workers checking AI-suggested greenhouse-gas figures taken from tables in corporate reports. Reviewers caught 82% of spelling-type errors and about 77% of wrong-year or wrong-scope errors, but only 31% of the errors that needed knowledge of the extraction rules. Requiring reviewers to type the correct value when they flagged an error led them to flag fewer, and participants favourable towards AI caught fewer mistakes.

In vision the equivalent is a mask that follows the wrong edge, or a small object the detector never proposed. The missed object is the worst case, because a verify-only queue has nothing to show the reviewer. Four countermeasures are cheap:

• Write the model's known error types, with examples, into the annotation guidelines.
• Seed pre-labels you know are wrong into the queue and measure how many are caught.
• Keep a slice labelled from scratch by a different annotator and compare error profiles.
• Make “add a missing object” a logged action. If nobody ever uses it, the detector is setting your recall ceiling.

Where foundation models fail

Each model paper lists its own failure cases, and they are the cases where pre-labels cost most to fix. The SAM authors write that it can miss fine structures and hallucinate small disconnected components. The SAM 2 appendix adds shot changes, crowded scenes and thin or fine details on fast-moving objects.

Domain shift is the larger effect. Mazurowski et al. (2023) tested SAM on 19 medical imaging datasets. With a single prompt, IoU ranged from 0.1135 for spine MRI to 0.8650 for hip X-ray; box prompts did notably better than points, and additional clicks helped only slightly. How the model behaves on your images is an empirical question that its general reputation does not answer.

Open-vocabulary detectors, which supply the “what” in text-prompted pipelines, show the same gap. Grounding DINO reports 52.5 AP zero-shot on COCO, whose everyday classes resemble its training data, and a mean of 26.1 AP zero-shot on the ODinW collection of real-world detection datasets. An empirical study of Grounding DINO feeding SAM as an automatic annotator, by Mumuni and Mumuni (2024), found that false positives often carried high confidence scores, so a confidence threshold alone did not separate good boxes from bad. Filtering out boxes of implausible size raised Dice substantially on several of its datasets.

Rare classes fail in a particular way: the model proposes them less often, and in a pre-label queue an absent box is easier to overlook than a wrong one. If a rare class matters, review the images where it should appear, not only the boxes the model drew. When the model is the problem rather than the workflow, our post on fine-tuning segmentation models covers when adapting it is worth the masks it needs.

A two-arm pilot before you commit

The only speed-up that matters is the one for your data, your model and your annotators, and a pilot of a few hundred items will measure it.

1. Draw a random sample and have it labelled from scratch by two senior annotators who then adjudicate their differences. This is the reference.
2. Split a second random sample into two arms: labelled from scratch, and pre-labelled then corrected. Same annotators, same guidelines, order balanced so neither arm always comes second.
3. Log time per item and the fate of every pre-label: accepted, edited, deleted, or an object added that the model missed.
4. Score both arms against the reference with the metric your model will be judged on, per class, with recall reported separately from precision.
5. Compare the errors. If the pre-labelled arm's mistakes match the model's mistakes more often than the scratch arm's do, that is anchoring, and an aggregate score will not show it.
6. Decide on cost per correct label, and repeat the pilot when the model or the domain changes. For categorical labels, Cohen’s kappa between each arm and the reference is a compact summary; for boxes and masks, use IoU.

Questions people ask