Promptable segmentation models are unusual: the most valuable thing you can do with one is often not to fine-tune it at all, but to use it to produce the training data for something else.
Consider not fine-tuning
A general promptable segmentation model will produce a high-quality mask for almost anything you point at, given a click, a box or a rough region. What it will not do is decide what to segment, because it has no concept of your classes. (Meta's SAM 3 is the exception: it also takes a short noun phrase or image exemplar and segments every instance of that concept, so test it first.)
That division is the key to using these models well. If your problem is “produce an accurate mask for this thing I have located”, the pretrained model very likely already does it, and fine-tuning is wasted effort. If your problem is “find all the instances of my class and segment them”, you need something that supplies the what — a detector, a classifier, or a prompt — and the segmentation part may need no training whatsoever.
So the most productive pattern is frequently: a cheap detector finds your objects, the segmentation model turns each box into a precise mask, and you have an instance segmentation pipeline without ever annotating a mask by hand.
When fine-tuning is genuinely warranted
Two cases, both involving domain shift away from natural photographs.
Unusual imagery. Medical scans, electron microscopy, satellite imagery, infrared, industrial radiography. A model pretrained on web photographs has no notion of what an object boundary looks like in these, and mask quality degrades accordingly.
Boundary conventions your domain defines. Where exactly does a tumour end, or a weld, or a cloud? These are domain judgements rather than visual edges, and a general model will give you a plausible boundary that is not the one your field uses. This is often the stronger reason to fine-tune, and the one people notice late.
In both cases you need ground-truth masks drawn to your convention, and comparatively few of them — you are adjusting a capable model's notion of boundaries, not teaching it segmentation from scratch.
The data shape, and the prompts
Training data for a promptable model is images, ground-truth masks, and the prompts that should produce those masks. That third element is the part that differs from conventional segmentation training.
Prompts can be points, boxes or coarse masks (and, for SAM 3, short text phrases or image exemplars). During training they are typically sampled — a point near the object centre, a box around it, sometimes with jitter so the model does not learn to depend on perfectly placed prompts. The sampling strategy should resemble how prompts will arrive at inference: if a detector will supply slightly loose boxes, train with slightly loose boxes.
Mask quality is the ceiling, and it matters more here than in most tasks because precise boundaries are the entire product. Polygon approximations with a handful of vertices will teach the model to produce polygon approximations. If you are commissioning annotation, specify boundary tolerance explicitly and audit against it.
{"image": "scans/0412.png",
"mask": "masks/0412.png",
"prompt": {"type": "box", "box": [112, 64, 398, 301], "jitter": 0.05}}
# mask: binary, same dimensions as the image
# type: "box", or "point" / "points" / "coarse_mask"
# box: [x_min, y_min, x_max, y_max]
# jitter: perturb in training so the model tolerates imprecise
# prompts at inference
# sample prompts the way they will actually arrive:
# detector-supplied boxes -> train on loose, jittered boxes
# human clicks -> train on single points, varied placement
Evaluating segmentation properly
Intersection over union is the standard metric and it is systematically misleading in one direction: it is dominated by object area, so boundary accuracy on large objects barely moves it while being exactly what you fine-tuned for.
Report boundary-focused metrics alongside region overlap, and break results down by object size. A model that scores well overall may be poor on the small objects that matter in your application — and in COCO-derived evaluation the size buckets are computed from mask area, so if you populated that field from bounding boxes your breakdown is skewed. Our COCO format post covers that detail.
Then look at masks. Overlay predictions on a sample at full resolution and inspect the boundaries. Segmentation is the task where numeric metrics diverge most from perceived quality, and a few minutes of looking tells you things no aggregate will.
Questions people ask
Can I use a promptable model to annotate data automatically?
Yes, and this is its best use for most teams. Cheap boxes in, precise masks out, with human review of a sample. It converts the most expensive annotation task into one of the cheapest, and the resulting masks can train a small deployable model.
How many masks do I need to fine-tune?
Fewer than for training from scratch — hundreds rather than tens of thousands — because you are adjusting a capable model's boundary behaviour. Quality and consistency of boundary convention matter far more than count.
Does it work on 3D or volumetric data?
Not natively — these models are built for 2D images. The common approach is slice-by-slice segmentation with some form of propagation between slices, and domain-specific variants exist for volumetric medical data. Check for one before adapting a 2D model yourself.
Semantic or instance segmentation?
A promptable model naturally produces instances: one point or box prompt, one object, one mask (a SAM 3 text prompt returns every instance). For semantic segmentation you would merge masks by class, which works but is a roundabout route — a conventional semantic segmentation model is the more direct tool for per-pixel class labels.