Fine-tuning CLIP: contrastive pairs and the batch-size problem

Published 5 October 2026 · 5 min read

CLIP trains on image-text pairs with no labels at all — the supervision comes from matching the right caption to the right image within a batch. That one architectural fact determines everything about how you prepare data and what hardware you need.

The data is pairs, and the negatives are implicit

A contrastive dataset is simply images with their descriptions. There is no class list, no bounding box, no label taxonomy. One image, one piece of text that describes it.

The training objective is to match each image to its own caption and not to the other captions in the batch. So the negative examples are the other items in the batch, generated for free. That is what makes the approach scale to enormous datasets without annotation effort — and it is also the source of its main practical constraint.

Because the negatives come from the batch, batch size is not a throughput knob here: it is part of the objective. A batch of 64 asks the model to distinguish a caption from 63 alternatives; a batch of 32,768 asks it to distinguish one from tens of thousands. The larger task teaches far more. This is why the original contrastive results used very large batches, and why fine-tuning on a small GPU with a small batch gives weaker gains than the dataset size suggests.

Each image's negatives are the other captions in its batch: a batch of N gives it N−1 of them, and at CLIP's original batch size of 32,768 that is more than 32,000 per image.

Caption style decides what you get

The model learns the relationship between images and the kind of text you pair with them, so caption style is a design decision rather than a detail.

Short alt-text-style captions teach broad category-level association and suit zero-shot classification, where you will prompt with short class names. Longer descriptive captions teach finer attribute and relationship distinctions, and suit retrieval where queries are phrases rather than words.

The practical implication is that your captions should resemble your queries. If users will search your image collection with natural phrases, train on natural phrases. If you will use the model for zero-shot classification against a fixed label set, train on text that looks like those labels. Our captioning datasets post covers the caption-length bands in more detail.

Noisy captions are more tolerable here than in most training. Web-scraped pairs are substantially mismatched and contrastive learning still works, because the objective only needs the correct pair to be more similar than the random ones. Do not spend heavily on caption polish; spend on coverage of the concepts you care about.

Diagram of contrastive image-text pretraining and zero-shot classification. — Contrastive pretraining: the model matches each image to its own caption and not to the others in the batch, which is where the negatives come from. Image: openai/CLIP · MIT repository

When fine-tuning helps, and when it hurts

A pretrained contrastive model is already strong on general imagery, so the case for fine-tuning is domain shift: medical images, satellite imagery, industrial parts, product catalogues, anything whose vocabulary and appearance are unlike web photographs.

Two radar charts titled VL-PAW dataset and CNU Weed dataset with axes 0-shot, 1-shot, 2-shot, 3-shot, 10-shot and whole. The orange projection-only polygon encloses the blue full fine-tuning polygon on the few-shot axes, reaching values such as 72.82 and 80.0, and meets it at the whole-dataset axis near 90. — Accuracy of a vision–language model fine-tuned in full (blue) or only in its projection heads (orange) on two agricultural image sets, from 1 example per class to the whole training set. With few examples, tuning only the projection heads scores clearly higher; the gap closes once all the data is used. Training only part of the model is one way to stop fine-tuning from overwriting what pretraining learned. Image (top two panels, cropped): Figure 6 of the article · CC BY 4.0 · from Yu et al. 2025, Electronics, “VL-PAW: A Vision–Language Dataset for Pear, Apple and Weed”, doi:10.3390/electronics14102087

The risk is losing the general capability that made the model useful. Fine-tuning hard on a narrow domain degrades zero-shot performance elsewhere — a well-documented trade, and often an acceptable one if your application only ever sees the narrow domain.

If you need both, the mitigations are a low learning rate, a short run, mixing general pairs back in, or training only part of the model. And measure both: your domain metric and a general retrieval benchmark. A fine-tune that improves your domain by a little and destroys general performance is easy to ship unnoticed if you only measure the first.

the dataset, and the thing to measure

# a contrastive dataset is this, and nothing more
{"image": "parts/bearing_0412.jpg",
 "text":  "a worn radial ball bearing with pitting on the outer race"}

# train with the largest effective batch you can reach -
# the other items in the batch ARE the negatives.

# evaluate on BOTH, every time:
#   1. your domain task   (retrieval / zero-shot on your own held-out pairs)
#   2. a general benchmark (did you break what you started with?)

A schematic plot of out-of-distribution accuracy against in-distribution accuracy. Zero-shot CLIP models (purple stars) sit on a high line and models trained on the in-distribution set (blue dots) on a lower one; a dashed arrow runs from the best zero-shot model to 'Fine-tuned CLIP' (orange square), further right but lower, and green arrows mark 'effective robustness' as the height above the blue line. — The WiSE-FT authors' summary of what end-to-end fine-tuning does to CLIP: accuracy on the target distribution rises, and much of the robustness to shifted data that made the zero-shot model valuable is lost. Measure both before concluding that a fine-tune has helped. Image (cropped): mlfoundations/wise-ft · MIT repository · from “Robust fine-tuning of zero-shot models”, arXiv:2109.01903

Preparing the data

Deduplicate before training, by image and by caption. Contrastive training is unusually sensitive to duplicates because a duplicate image with a different caption, or the same caption on two images, puts contradictory pressure on the objective — the model is asked to treat a correct pairing as a negative.

Check for the extreme case too: captions that are identical across many images, which are common in scraped product data where every item has the same boilerplate description. Those carry no discriminative signal and actively confuse the objective.

On scale: contrastive pretraining is a large-data regime, but contrastive fine-tuning is not. Tens of thousands of good domain pairs can meaningfully shift a pretrained model, and that is a tractable collection job — particularly if you already have an image collection with text attached, which most organisations with a product catalogue or document archive do.

Line chart titled 'Scaling trends for zero-shot models': top-1 accuracy against the number of YFCC samples in millions on a log scale (0.5, 1, 3, 7, 15). The ImageNet line rises from about 0.03 to about 0.33 and the ImageNet V2 line from about 0.02 to about 0.28. — Zero-shot accuracy of CLIP models trained by the OpenCLIP team on 0.5 to 15 million YFCC pairs: ImageNet top-1 climbs from under 5% to about 33% and, in their words, is far from saturated at 15 million. That is the cost of pretraining; a fine-tune starts from a model that has already paid it. Image: mlfoundations/open_clip · MIT repository

Questions people ask

What batch size do I actually need?

Larger is better and the relationship is strong, because batch size sets the difficulty of the objective. If you are memory-limited, use gradient caching or distributed negatives rather than accepting a small batch — and be aware that results from a small-batch run understate what the data could achieve.

Can I fine-tune on single-word labels?

You can, by templating them into short phrases, and it works for zero-shot classification. You lose the attribute and relationship learning that longer captions provide, so if you have richer text available, use it.

How do I stop it forgetting general knowledge?

Low learning rate, short run, mix general pairs into the training data, or freeze part of the model. Most importantly, measure general performance as well as domain performance so you can see the trade rather than discovering it later.

Is a contrastive model the right choice for my search problem?

If you need text-to-image or image-to-image search over your own collection, yes — it is the standard and effective approach. If you need to answer questions about images, you want a vision-language generative model instead; contrastive models score similarity, they do not produce text.

Keep reading

MultimodalThe best multimodal datasets for vision-language models MultimodalVisual question answering datasets MultimodalImage captioning datasets