Caption datasets differ mainly in how detailed their descriptions are, and a model reproduces the length and style of whatever it was trained on.
Three bands of caption, suiting different purposes. Short alt-text-style captions of a few words suit contrastive pretraining, where you want a broad association between images and language. Sentence-length human captions suit caption generation in the conventional sense. Paragraph-length dense descriptions — increasingly generated by a strong vision-language model rather than written by people — suit training models that describe images thoroughly.
The practical point is that the model copies the distribution. Train on short captions and you get short captions, regardless of what you ask for at inference. Decide what output length you want before picking the data. And note that for the dense-caption sets, the descriptions are usually machine-generated, which means you inherit the generating model's errors and its characteristic phrasing along with its thoroughness.
| Dataset | Modality | Size | Licence | Best for |
|---|---|---|---|---|
| Blip3 Kale | Multimodal | 218M pairs | Apache 2.0 | Large vision-language models |
| Moondream2 COYO 5M Captions | Image | 5M pairs | CC BY-SA 4.0 | Text-to-image generation |
| WIT Base | Multimodal | 6.5M examples | CC BY-SA 3.0 | Combine images and text |
| LAION Art EN Improved Captions | Image | 2.68M pairs | CC BY 4.0 | Text-to-image generators |
| HumanCaption 10M | Image | 10M images | Research only | Natural captions for human images |
| Flickr Image Dataset | Multimodal | 31,800 images | Research only | Image captioning |
| BLIP3o Long Caption Dataset | Multimodal | 27M images | Apache 2.0 | Automatic visual description generation |
| 220k GPT4Vision Captions from LIVIS | Multimodal | 218,000 captions | Apache 2.0 | Image description generation |
| New Yorker Caption Contest | Multimodal | 10,900 instances | CC BY 4.0 | Understand multimodal humour |
Blip3 Kale
- Size
- Approximately 218 million image-text pairs, Parquet format, 6.87 GB per split
- Licence
- Apache 2.0
About 218 million image-caption pairs in Parquet at 6.87 GB per split, where captions are dense and enriched with web-sourced knowledge rather than being short alt-text. Salesforce-published and Apache licensed.
Pre-training large vision-language models, improving factual and detailed image description generation, and research into knowledge-enriched image understanding.
Source: innovatiana.com
Moondream2 COYO 5M Captions
- Size
- 5 million image-text pairs, 1.5 GB, image and text formats (captions)
- Licence
- CC-BY-SA 4.0
Five million image-text pairs totalling 1.5 GB, extracted from COYO-700M and recaptioned with long descriptions by the Moondream2 vision-language model, prompted with each image's alt text, from images pre-filtered for aesthetic score, watermark score and CLIP similarity. Hosted on ModelScope.
Training text-to-image generation, developing automatic image annotation and understanding, and improving long caption quality in multimodal applications.
Source: innovatiana.com
WIT Base
- Size
- About 6.5 million examples (6,477,255; the dataset viewer previews only the first 5.15 GB, about 108,000 rows), multilingual images and captions, Parquet format
- Licence
- CC BY-SA (upstream WIT is CC BY-SA 3.0)
About 6.5 million Wikipedia image-and-caption examples in Parquet (the viewer previews only the first 5.15 GB, about 108,000 rows), with captions in many languages. Wikimedia provenance makes it one of the cleanest multilingual image-text sources available.
Training models that combine images and text across several languages, improving multimodal translation and image description, and testing robustness on real-world data.
Source: innovatiana.com
LAION Art EN Improved Captions
- Size
- 2.68 million pairs of artistic images and improved captions, 442 MB, Parquet format
- Licence
- CC-BY 4.0
2.68 million artistic image-caption pairs in a 442 MB Parquet file, where the original alt-text has been replaced by improved English captions generated by a stronger model to tighten image-text alignment.
Fine-tuning text-to-image generators such as Stable Diffusion, building searchable prompt databases, and improving semantic alignment between images and descriptions.
Source: innovatiana.com
HumanCaption 10M
- Size
- Approximately 10 million human images with text descriptions, standard image formats, associated metadata
- Licence
- Tagged CC BY 4.0, but the card limits use to research and education (images from LAION-Face)
Roughly ten million human-centred photographs in standard image formats with accompanying metadata, each paired with a natural-language caption. Built for face and person-focused captioning rather than general scenes. The images come from LAION-Face, and despite the CC BY 4.0 tag the card limits use to research and education.
Generating natural captions for human images, improving facial recognition with contextual descriptions, and developing applications analysing people and human interaction.
Source: innovatiana.com
Flickr Image Dataset
- Size
- 31,800 images, 158,000 captions, 276,000 bounding boxes — JPEG, CSV
- Licence
- Flickr30k terms: non-commercial research and education; images under the Flickr Terms of Use (this mirror labels it CC0)
31,800 Flickr JPEG images with 158,000 captions and 276,000 bounding boxes in CSV, plus annotated entities and coreference chains linking caption phrases to image regions. The grounding annotations set it apart from plain captioning sets. Despite this mirror's CC0 tag, the images remain their Flickr owners' copyright and the original release is for non-commercial research and education.
Image captioning, aligning text entities with visual regions for grounded NLP, and testing joint image-and-language understanding.
Source: innovatiana.com
BLIP3o Long Caption Dataset
- Size
- Approximately 27 million images with captions (~120 tokens), WebDataset tar shards (Hugging Face also serves an auto-converted Parquet copy)
- Licence
- Apache 2.0
Roughly 27 million images in WebDataset tar shards, each paired with a lengthy AI-generated description of about 120 tokens. Caption length is the distinguishing feature versus short alt-text corpora.
Training automatic visual description generation, pre-training multimodal systems on complex images, and cross-modal search between long text and images.
Source: innovatiana.com
220k GPT4Vision Captions from LIVIS
- Size
- About 218,000 captions with LVIS image references, 62.8 MB Parquet
- Licence
- Apache 2.0
Roughly 218,000 GPT-4 Vision captions for LVIS images in a 62.8 MB Parquet file: the captions and image references, not the images themselves. A compact recaptioning resource published by LAION.
Training image description generation, improving multimodal contextual understanding, and testing a model's ability to summarise and formulate accurate captions.
Source: innovatiana.com
New Yorker Caption Contest
- Size
- About 10,900 matching, 10,600 ranking and 2,600 explanation instances, each repeated over five cross-validation splits; 7.91 GB, Parquet
- Licence
- CC-BY 4.0
Matching, ranking and explanation tasks (about 10,900, 10,600 and 2,600 instances, each repeated over five cross-validation splits) from the New Yorker caption contest in 7.91 GB of Parquet, with many human-written humorous captions per cartoon. A rare benchmark for humour, which demands cultural and pragmatic inference rather than description.
Testing and training models to understand multimodal humour, improving generation of creative captions, and exploring complex contextual understanding between images and text.
Source: innovatiana.com
How this list was put together
Found by searching Dataset Finder for datasets whose documentation describes the task, then checked by hand. A keyword match is not a fit — the first pass picked up a coral survey for the depth-estimation list because it was recorded below fifty metres — so every entry was read against what this page covers, and the ones that matched a word without matching the task were removed.
Sizes and licences come from each dataset's own documentation, vague wording included. Where we found a re-upload or catalogue card stating a licence that the original release contradicts — a CC0 tag on Flickr photographs, MIT on data whose terms require a signed agreement — the entry gives the original terms and says so. The comparison table abbreviates licences; hover a cell for the full wording.