Boxes, polygons, masks or keypoints: what each costs and buys

Seconds per label, as the papers measured them

The shapes themselves are well described elsewhere; Innovatiana’s data annotation guide is a good primer on the definitions. This article starts where such guides stop: how long each type took people to produce when a research team timed it, and what a model trained on it gained.

The numbers below come from studies that recorded annotator time with a fixed interface. They are not interchangeable to the second, because images, objects and tools differ, but the ratios between them hold up. The COCO paper gives its mask cost as more than 22 worker-hours per 1,000 segmentations, which is about 79 seconds each.

Keypoints, the type nobody timed

None of the timing studies above covers keypoints, and we have not found a comparable measurement. Their cost is driven by two things the other types lack: a fixed list of named points per instance (17 for a COCO person), and a visibility flag on each, which forces a judgement on every occluded joint. Skeleton definitions also differ between datasets, as our pose estimation datasets post explains, so a keypoint order has to name the skeleton as well as the type.

What has been measured is how few keypoint labels a model can need. The DeepLabCut authors found that with about 200 labelled frames their network tracked animal body parts with accuracy comparable to a human, and around 100 frames gave errors of 5 pixels or less. Human labelling variability on the same task was 2.69 pixels. That is a controlled laboratory setting, but it suggests the useful question for keypoints is how many frames rather than how many seconds per point.

For budgeting, price keypoints per point rather than per person, and allow for the occlusion decisions. Crowded scenes cost more per point than isolated subjects, because more joints are hidden and each one still needs a flag.

The cheap end: tags, points and scribbles

The What’s the Point study by Bearman et al. timed annotators on PASCAL VOC 2012 and trained segmentation models from each kind of label. Image-level tags alone gave 29.8% mIoU. One point per class, which added about two seconds per image, gave 35.1%, and 42.7% with an objectness prior. Under a fixed annotation budget, point-trained models beat image-level, squiggle and full supervision by 2.7 to 20.8 points of mIoU.

The reason is arithmetic. At 239.7 seconds per fully annotated image, the time that buys one full image buys about ten point-annotated ones, and for that model and dataset breadth beat precision.

Points work for detection too. In the click-supervision study by Papadopoulos et al., a centre click took 1.87 seconds. A detector trained from one click per object reached 45.9% mAP on PASCAL VOC 2007, against 55.5% with full boxes. Two clicks per object reached 49.1% for 7.6 hours of annotation, against 71 hours for the boxes.

For instance masks, Cheng, Parkhi and Kirillov had annotators mark ten random points inside each existing box as object or background. That was about five times faster than drawing full masks, and Mask R-CNN trained on it reached 94 to 98% of its fully supervised performance on COCO, PASCAL VOC, Cityscapes and LVIS.

How much accuracy each extra second buys

Boxes are the clearest case of paying less for the same thing. With extreme clicking, annotators click the top, bottom, left and right extremes of the object and the box follows. Detectors trained on those boxes matched detectors trained on the original PASCAL boxes exactly (56 mAP with AlexNet and 66 with VGG16 on VOC 2007), and 91 to 94% of the boxes overlapped the ground truth at IoU above 0.7.

For segmentation, the most direct comparison is by Zhang et al. (2023), who tested six annotation styles on two natural-image and two medical datasets under fixed budgets. Their conclusion is blunt: precisely outlining objects is “virtually never the optimal approach when annotation budget is limited”. Above roughly 0.8 mIoU, label quality had little effect on the trained model. Boxes followed by zero-shot segmentation with SAM were among the most cost-effective options, and scribbles worked well on natural images but poorly on CT and MRI.

Precision still pays in three places: when the output is a measurement taken from the mask (an area, a volume, a defect size), when objects are thin or interlocking so that a box covers mostly background, and in the test split. A coarse test mask caps the score you can measure, so keep the evaluation set precise whatever you buy for training.

There is also a ceiling on what precision can buy, and people set it. When 30 Cityscapes images were annotated twice by different annotators, 96% of pixels received the same label, or 98% excluding void. The Segment Anything authors cite prior estimates of inter-annotator mask consistency at 85 to 91% IoU. Paying for boundary accuracy beyond the point where two careful annotators agree buys noise.

Converting between types, and what you lose

Conversion is free in one direction. A mask gives you its box exactly, a polygon rasterises to a mask, and either gives a centre point and an image-level tag. Going the other way needs a model, and the model’s errors become label errors.

• Mask or polygon to box: lossless. Take the extremes of the pixels or vertices.
• Polygon to mask: lossless at the polygon’s own precision. Holes and multi-part objects depend on the format; our COCO JSON format post covers its two segmentation encodings.
• Box to mask: needs a segmenter. In the extreme-clicking paper, masks made from extreme points with GrabCut matched the ground truth at 78.1% mIoU on VOC 2007, 3.7 points above GrabCut masks made from boxes alone. A box-prompted SAM is the current equivalent; review a sample before trusting it.
• Points to boxes: not without a model, because a point has no extent.
• Anything to keypoints: not possible. A keypoint is a named position, such as a left wrist, that no outline encodes.

Choosing by task, and pricing the order

Start from what the model must output, then pick the cheapest label that supervises it. Write the chosen type and its rules (tightness, occlusion, minimum size) into the guidelines before work starts, because changing type midway means relabelling; our annotation guidelines template produces the skeleton.

The table below prices one order, 10,000 images with seven objects each, from the measured times. It counts annotation only, plus verification where the source included it. Our free annotation cost estimator runs the same arithmetic with your own counts.

• Image class: image-level tags. Add points only if small objects in multi-label images get missed.
• Counting or locating: one click per object. Move to boxes when objects overlap and extents matter.
• Detection: extreme-click boxes. Polygons add nothing a box detector can use, unless objects are thin or rotated and you need tight oriented boxes derived from them.
• Instance masks: boxes plus a box-prompted segmenter with review, or points inside boxes; polygons where the mask itself is measured. Our instance segmentation post covers what that data must contain.
• Semantic segmentation: points or coarse polygons for training, fine masks for the test split. On CT and MRI, Zhang et al. found scribbles underperformed.
• Pose: keypoints with visibility flags. There is no cheaper substitute, so spend the effort on the number of frames.
• Mixed: rich labels for a few thousand images and cheap ones for the rest. Cityscapes shipped 5,000 fine and 20,000 coarse images, and 97% of labelled pixels in the coarse set matched the fine labels.

Questions people ask