Document AI is four different tasks wearing one name, and picking a dataset for the wrong one is the most common mistake in the category.
The four tasks, in increasing difficulty. Text recognition reads characters from an image. Layout analysis identifies regions — headings, paragraphs, tables, figures — and their reading order. Table extraction recovers the row and column structure of tabular content, which is much harder than reading the cells. Form and document understanding answers questions about a document, which requires all three plus reasoning.
Modern recognition is largely solved for clean printed text, so the datasets that matter are the ones covering what is not clean: handwriting, historical documents, low-resolution scans, photographs taken at an angle, multi-column layouts, and documents in scripts other than Latin. If your documents are clean printed English, you may not need to train anything at all — which is worth testing before you collect data.
| Dataset | Modality | Size | Licence | Best for |
|---|---|---|---|---|
| OlmoCR PES2O Dataset | Text | 7.9M papers | ODC-By | Language models on scientific text |
| OGC MEGA MultiDomain DocRetrieval | Image | 1.09M examples | Apache 2.0 | Visual document retrieval models |
| OCRFlux Bench Single | Text | 2,000 pages | Apache 2.0 | OCR accuracy on complex layouts |
| Text Extraction for OCR | Multimodal | 1,560 images | CC0 | Invoice-specific entity extraction |
| OCR Benchmark | Multimodal | 1,000 examples | MIT | OCR accuracy of multimodal models |
| BLIP3 OCR 200M | Multimodal | 96M rows | Research only | VLMs on text-heavy images |
| TextOCR: Extraction of text on natural images | Image | 28,134 images | CC BY 4.0 | OCR on complex text |
| CharXiv: Understanding Scientific Graphics | Multimodal | 2,323 images | CC BY-SA 4.0 | Visual understanding of multimodal models |
| Sujet Finance QA Vision 100k | Multimodal | 107,050 pairs | Apache 2.0 | Visual question answering for complex documents |
OlmoCR PES2O Dataset
- Size
- About 7.9 million academic papers as extracted text with metadata (111 GB)
- Licence
- ODC-BY
Academic papers from the peS2o collection, re-extracted to plain text with olmOCR and released by AllenAI: each record holds the paper's text, page-level character offsets and metadata such as page and token counts. It is text, not page images, so it serves language-model training on scientific writing rather than training an OCR model.
Pretraining language models on scientific text, building retrieval over papers, and comparing OCR-extracted text quality.
Source: innovatiana.com
OGC MEGA MultiDomain DocRetrieval
- Size
- Around 1.09 million examples with images, query texts, languages (fr, en, de, es, it), JSON formats + PNG/JPEG images
- Licence
- Apache 2.0
About 1.09 million examples merging seven sources, each with a text query, positive images and negative image sets, across French, English, German, Spanish and Italian. Covers military, energy and geotechnical document domains.
Training visual document retrieval models, applying contrastive learning to improve search relevance, and strengthening robustness on multilingual documents.
Source: innovatiana.com
OCRFlux Bench Single
- Size
- 2,000 annotated pages (Markdown), bilingual EN/ZH
- Licence
- Apache 2.0
2,000 PDF pages with manual Markdown annotations, bilingual across English and Chinese, for evaluating how accurately OCR systems parse complex layouts including titles, tables and structured text into Markdown output.
Measuring OCR accuracy on complex layouts, testing multilingual OCR on English and Chinese, and improving Markdown output from scanned or PDF images.
Source: innovatiana.com
Text Extraction for OCR
- Size
- About 1,560 invoice images (JPG/PNG) paired with XML entity annotations, including extracted tabular data
- Licence
- CC0: Public Domain
About 1,560 pairs of invoice images in JPG or PNG with XML files holding extracted entities such as invoice number, date, business name, telephone and address, plus extracted tabular data. A ready-made document-NER set.
Developing invoice-specific entity extraction, improving recognition of tabular data in complex images, and building OCR robust to typographic and quality errors.
Source: innovatiana.com
OCR Benchmark
- Size
- 1,000 examples, about 386 MB, Parquet and JSON format
- Licence
- MIT
A 1,000-example benchmark of about 386 MB in Parquet and JSON, for comparing how well multimodal LLMs perform OCR and extract structured JSON from documents. Evaluation-sized rather than training-sized.
Measuring OCR accuracy of multimodal models, comparing structured JSON extraction quality across LLMs, and improving joint visual and text comprehension.
Source: innovatiana.com
BLIP3 OCR 200M
- Size
- About 96 million rows in Parquet (image URLs with OCR text and boxes); the BLIP-3 paper describes 200 million images
- Licence
- Apache 2.0, but released for research purposes only; records hold image URLs and OCR metadata, not the images
Roughly 96 million Parquet rows of OCR annotations for web images that the records reference by URL rather than include, built to help vision-language models read text-heavy visuals such as documents, diagrams and screenshots. Its Hugging Face release shows about 96 million rows, against the 200 million images the BLIP-3 paper describes (the card itself says about 2 million samples), so check what a row holds before quoting a size. Salesforce releases it for research purposes only.
Pre-training VLMs on text-heavy images, raising OCR accuracy on real-world documents, and boosting multimodal reasoning.
Source: innovatiana.com
TextOCR: Extraction of text on natural images
- Size
- 28,134 images, about 900,000 word annotations, JPEG and JSON
- Licence
- Annotations CC BY 4.0; images CC BY 2.0 (Open Images, via TextVQA)
28,134 natural-scene images from TextVQA with about 900,000 word-level polygon annotations in JSON. Polygon rather than box annotation lets it cover curved and partially occluded text. The annotations are CC BY 4.0 and the Open Images photos CC BY 2.0, so attribution is required.
Training OCR on complex text such as curved or partially visible words, improving VQA or multimodal captioning, and testing robustness across fonts and backgrounds.
Source: innovatiana.com
CharXiv: Understanding Scientific Graphics
- Size
- 2,323 high resolution images with 11,000+ questions, Parquet format
- Licence
- CC BY-SA 4.0 for the questions and metadata; chart images remain the copyright of the original arXiv authors
2,323 high-resolution scientific charts taken from arXiv papers, paired with more than 11,000 open-ended questions in Parquet. Real published figures rather than synthetic plots, which makes it considerably harder than chart-QA predecessors. The CC BY-SA 4.0 licence covers the questions and metadata only; the chart images remain the copyright of the original authors.
Evaluating visual understanding of multimodal models, testing robustness on open QA over realistic scientific data, and training models for chart reading.
Source: innovatiana.com
Sujet Finance QA Vision 100k
- Size
- 107,050 QA pairs, 9,801 images, structured format (JSON/image)
- Licence
- Apache-2.0
107,050 English question-answer pairs generated from 9,801 images of financial documents such as reports, tables and invoices, in a structured JSON and image format. Professional document VQA rather than natural scenes.
Developing visual question answering for complex documents, improving financial document analysis via OCR and semantic understanding, and refining multimodal LLMs.
Source: innovatiana.com
- title
- Few-Shot Class-Incremental Learning
- paper_id
- 2004.10956
- caption
- Comparison results under the \emph{5-way 10-shot} and \emph{5-way full-shot} settings, evaluated with ResNet18 on miniImageNet.
- query
- Which model shows a greater decline in accuracy from Session 1 to Session 9 in the 5-way full-shot scenario?
- answer
- Joint-CNN
How this list was put together
Drawn from the datasets in Dataset Finder filed under this topic, then checked by hand against what this page covers. A tag is a filing decision, not a description, so records that carried the tag without being about the topic were removed, and relevant records filed elsewhere were added. The order puts the closest fit first.
Sizes and licences come from each dataset's own documentation, vague wording included. Where we found a re-upload or catalogue card stating a licence that the original release contradicts — a CC0 tag on Flickr photographs, MIT on data whose terms require a signed agreement — the entry gives the original terms and says so. The comparison table abbreviates licences; hover a cell for the full wording.