The COCO JSON format, explained field by field

The five top-level keys

A COCO annotation file is a single JSON object. Only three of its keys carry data you need: images, annotations and categories. The other two, info and licenses, are metadata that most loaders ignore entirely — though you should fill them in, because they are the only place the provenance of a derived dataset survives.

The structure is relational, not nested. Images do not contain their annotations. Instead every annotation carries an image_id that points back at an entry in images, and a category_id that points at categories. That is why a COCO file stays easy to join and validate: the three lists are independent and linked only by ids, though most loaders still parse the whole document into memory.

{
  "info": {"description": "...", "version": "1.0", "year": 2026},
  "licenses": [{"id": 1, "name": "CC BY 4.0", "url": "..."}],
  "images": [
    {"id": 1, "file_name": "000001.jpg", "width": 640, "height": 480, "license": 1}
  ],
  "annotations": [
    {"id": 1, "image_id": 1, "category_id": 3,
     "bbox": [x, y, width, height], "area": 1234.5,
     "segmentation": [[x1,y1, x2,y2, ...]], "iscrowd": 0}
  ],
  "categories": [
    {"id": 3, "name": "car", "supercategory": "vehicle"}
  ]
}

bbox is not what most people assume

The single most common error in hand-written COCO files is the bounding box convention. COCO's bbox is [x_min, y_min, width, height] in absolute pixels, with the origin at the top left of the image.

It is not [x_min, y_min, x_max, y_max] — that is Pascal VOC. It is not normalised to 0–1 — that is YOLO. And it is not centre-based — that is YOLO too. Converting between them is three lines of arithmetic, but getting the direction wrong produces boxes that look plausible on small objects and wildly wrong on large ones, which is exactly the failure mode that survives a quick eyeball check and ruins a training run.

segmentation has two different shapes

The segmentation field is polymorphic, and which shape you get depends on iscrowd. This catches people who write a loader against one half of a dataset and then hit the other half.

When iscrowd is 0, segmentation is a list of polygons, each a flat list of alternating x and y coordinates: [[x1, y1, x2, y2, ...]]. An object split by occlusion gets several polygons in that outer list.

When iscrowd is 1, the object is a crowd region — a pile of bicycles, a stadium of people — and segmentation becomes a run-length encoding object instead: {"counts": ..., "size": [h, w]}. You need pycocotools to decode it. Models are conventionally not penalised on crowd regions during evaluation, which is the entire reason the flag exists.

area, and why it decides your metrics

area is the pixel area of the segmentation mask, not of the bounding box. For a diagonal object the two differ by a lot, and COCO evaluation buckets objects into small (area < 32²), medium and large (area > 96²) using this field.

If you populate area from the bbox because you have no masks, your per-size breakdown shifts and your numbers stop being comparable to published results on the same dataset. That is acceptable as long as you say so; it is a silent problem if you do not.

Category ids are not zero-indexed, and that is contagious

In the original MS COCO release the 80 categories carry ids running up to 90, with gaps — the ids are historical and not contiguous. Several frameworks silently remap them to a dense 0–79 range internally, and several do not.

The practical consequence: a class index printed by your training loop may not be the category_id in your file. Before you debug a model that seems to confuse two classes, check whether you are looking at a remapping artefact. Keep an explicit id-to-name mapping next to your checkpoint and you never have to ask the question twice.

import json, collections

d = json.load(open("annotations.json"))
imgs = {i["id"]: i for i in d["images"]}
cats = {c["id"] for c in d["categories"]}

bad = collections.Counter()
for a in d["annotations"]:
    im = imgs.get(a["image_id"])
    if im is None:                      bad["orphan annotation"] += 1; continue
    if a["category_id"] not in cats:    bad["unknown category"]  += 1
    x, y, w, h = a["bbox"]
    if w <= 0 or h <= 0:                bad["degenerate box"]    += 1
    if x < 0 or y < 0:                  bad["negative origin"]   += 1
    if x + w > im["width"] + 1 or y + h > im["height"] + 1:
                                        bad["box outside image"] += 1

print(f"{len(d['annotations'])} annotations, {len(d['images'])} images")
print(dict(bad) or "clean")

Questions people ask