VQA is the standard instruction-tuning task for vision-language models, and its benchmarks have a well-documented weakness worth understanding.
The language-prior problem is this: many VQA questions can be answered from the question alone, because the world is predictable. Asked what colour a banana is, a model that never looks at the image will usually be right. Early benchmarks were substantially answerable this way, and models scored well without visual grounding.
Later datasets address it by balancing — ensuring each question appears with images having different answers, so guessing from the question fails. When evaluating a VQA dataset, check whether it was balanced this way, and whether anyone has reported a blind baseline: the accuracy of a model given only the questions. The gap between that and your model's score is the part that is actually about seeing.
| Dataset | Modality | Size | Licence | Best for |
|---|---|---|---|---|
| The Cauldron | Multimodal | 1.88M entries | Per sub-dataset | Vision-language models on image-text tasks |
| MMPR v1.2 | Multimodal | 3M samples | MIT | Visual comprehension combined with reasoning |
| CV-Bench | Multimodal | 2,638 examples | Apache 2.0 | Assessing 2D and 3D abilities |
| MathVision | Image | 3,040 problems | MIT | Visual mathematical reasoning in multimodal models |
| Cambrian Alignment Dataset | Multimodal | 291,750 rows | Apache 2.0 | Aligning vision-language models for contextual understanding |
| VQA-RAD: Medical Imaging Questions and Answers | Medical | 2,244 pairs | CC0 | Specialised medical VQA models |
| SLAKE | Medical | 14,028 pairs | CC BY 4.0 | Answer questions about medical images |
| PathVQA - Medical questions on images | Multimodal | 32,632 pairs | Unclear — see below | Specialised medical imaging VQA models |
| PubMedVision | Medical | 1.3M pairs | Apache 2.0 | Multimodal medical VQA models |
| Sujet Finance QA Vision 100k | Multimodal | 107,050 pairs | Apache 2.0 | Visual question answering for complex documents |
The Cauldron
- Size
- 1.88 million entries, 169 GB, Parquet formats (images and text)
- Licence
- CC BY 4.0 for the prompts only; each of the 50 sub-datasets keeps its own licence
1.88 million entries across 169 GB of Parquet, unifying 50 separate vision-language training sets into one schema. Built to fine-tune the Idefics2 model, it saves considerable work assembling a mixed instruction corpus. CC BY 4.0 covers only the prompts: each of the 50 sub-datasets keeps its own licence.
Training vision-language models on image-text tasks, improving captioning, VQA, OCR and visual reasoning, and pre-training open-source architectures.
Source: innovatiana.com
MMPR v1.2
- Size
- About 3 million preference samples (image, question, chosen and rejected response), 54.7 GB
- Licence
- MIT
A multimodal reasoning preference dataset of about 3 million samples (54.7 GB), each an image and question with a chosen and a rejected response, used for Mixed Preference Optimization of InternVL models and reported against benchmarks such as MathVista, MathVision and POPE.
Improving visual comprehension combined with reasoning, training models for complex VQA, and testing robustness against visual hallucinations.
Source: innovatiana.com
CV-Bench
- Size
- 2,638 examples with 2D/3D annotations and images, 810 MB, Parquet and JSONL
- Licence
- Apache 2.0
2,638 manually inspected examples with 2D and 3D annotations plus associated images, 810 MB shipped as both Parquet and JSONL, drawing labels from ADE20k, COCO and OMNI3D. It tests whether vision-language models grasp spatial layout and depth.
Assessing 2D and 3D abilities of multimodal models, probing comprehension of spatial layout and depth, and benchmarking vision and multimodal research.
Source: innovatiana.com
MathVision
- Size
- 3,040 math problems with images (3,344 rows including the 304-problem testmini split), 64 MB, Parquet format
- Licence
- MIT
3,040 visual mathematics problems (plus a 304-problem testmini split) in a 64 MB Parquet file, drawn from real competitions and each paired with a contextual image, spanning 16 disciplines and five difficulty levels. Competition provenance keeps difficulty high.
Evaluating visual mathematical reasoning in multimodal models, training models to solve problems using images and text, and testing across mathematical disciplines.
Source: innovatiana.com
Cambrian Alignment Dataset
- Size
- 291,750 rows, over 50 GB, files archived in tar
- Licence
- Apache 2.0
291,750 rows across more than 50 GB of tar archives, holding question-answer pairs for aligning multimodal text-and-image models, drawn together from LLaVA, Mini-Gemini, ALLaVA and ShareGPT4V.
Training and aligning vision-language models for contextual understanding, evaluating LLM response quality on multimodal tasks, and creating robust multimodal benchmarks.
Source: innovatiana.com
VQA-RAD: Medical Imaging Questions and Answers
- Size
- 2,244 QA pairs over 314 radiology images (CT, MRI, X-ray), JSON
- Licence
- CC0 1.0
A radiology visual question answering set in which clinicians wrote both open-ended and yes/no questions about 314 radiology images (head CT and MRI, chest X-rays and abdominal CT), producing 2,244 question-answer pairs in JSON. Small but clinician-authored, it is the standard starting point for medical VQA work.
Training specialised medical VQA models, evaluating AI comprehension of medical images, and supporting AI-assisted diagnostic research.
Source: innovatiana.com
SLAKE
- Size
- 14,028 QA pairs over 642 images (CT, MRI, X-ray), JSON and Parquet, ~4.2 MB
- Licence
- CC-BY 4.0
A bilingual (English and Chinese) medical visual question answering set of 14,028 question-answer pairs over 642 images spanning CT, MRI and X-ray. Annotations come as JSON and Parquet, giving a compact resource for clinical image understanding.
Training models to answer questions about medical images, strengthening diagnostic support tools, and assessing AI visual understanding in clinical settings.
Source: innovatiana.com
PathVQA - Medical questions on images
- Size
- 32,632 QA pairs on 4,289 images, image + text formats
- Licence
- MIT on the Hugging Face release; rights in the textbook and digital-library images are not specified
32,632 question-answer pairs built on 4,289 pathology images taken from textbooks, mixing open-ended and yes/no questions. Textbook provenance makes it a standard medical VQA benchmark for histopathology, but the MIT tag covers the release, not the textbook images it reproduces.
Training specialised medical imaging VQA models, evaluating visual and textual understanding, and benchmarking visual QA in medicine.
Source: innovatiana.com
PubMedVision
- Size
- Approximately 1.3 million medical VQA pairs, 902 MB, Parquet format
- Licence
- Apache 2.0
Roughly 1.3 million medical image question-and-answer pairs built from PubMed image-text pairs that GPT-4V reformatted into question-answer form, delivered as Parquet. The scale and permissive licence make it one of the few medical multimodal corpora usable for large-scale pre-training rather than evaluation alone.
Training multimodal medical VQA models, improving understanding of medical images in context, and building assistants for healthcare professionals.
Source: innovatiana.com
Sujet Finance QA Vision 100k
- Size
- 107,050 QA pairs, 9,801 images, structured format (JSON/image)
- Licence
- Apache-2.0
107,050 English question-answer pairs generated from 9,801 images of financial documents such as reports, tables and invoices, in a structured JSON and image format. Professional document VQA rather than natural scenes.
Developing visual question answering for complex documents, improving financial document analysis via OCR and semantic understanding, and refining multimodal LLMs.
Source: innovatiana.com
How this list was put together
Found by searching Dataset Finder for datasets whose documentation describes the task, then checked by hand. A keyword match is not a fit — the first pass picked up a coral survey for the depth-estimation list because it was recorded below fifty metres — so every entry was read against what this page covers, and the ones that matched a word without matching the task were removed.
Sizes and licences come from each dataset's own documentation, vague wording included. Where we found a re-upload or catalogue card stating a licence that the original release contradicts — a CC0 tag on Flickr photographs, MIT on data whose terms require a signed agreement — the entry gives the original terms and says so. The comparison table abbreviates licences; hover a cell for the full wording.