Recognition of clean printed text is largely solved, so the datasets that matter are the ones covering everything that is not clean.
Before collecting data for this, test an off-the-shelf engine on your documents. If they are clean printed text in a Latin script, you may need no training data at all, and that is the cheapest possible outcome.
Where training is still needed: handwriting, which varies without limit between writers; historical documents with period typography, degradation and archaic orthography; scene text photographed at an angle under uncontrolled lighting; low-resolution scans and faxes; non-Latin scripts, particularly those with complex shaping or right-to-left ordering; and structured documents where recovering the layout matters as much as the characters. The sets below cover scene text, invoices and document parsing; handwriting beyond MNIST's isolated digits, and historical documents, need sources not on this list.
| Dataset | Modality | Size | Licence | Best for |
|---|---|---|---|---|
| Text Extraction for OCR | Multimodal | 1,560 images | CC0 | Invoice-specific entity extraction |
| OCRFlux Bench Single | Text | 2,000 pages | Apache 2.0 | OCR accuracy on complex layouts |
| OCR Benchmark | Multimodal | 1,000 examples | MIT | OCR accuracy of multimodal models |
| BLIP3 OCR 200M | Multimodal | 96M rows | Research only | VLMs on text-heavy images |
| TextOCR: Extraction of text on natural images | Image | 28,134 images | CC BY 4.0 | OCR on complex text |
| Sujet Finance QA Vision 100k | Multimodal | 107,050 pairs | Apache 2.0 | Visual question answering for complex documents |
| MNIST | Image | 70,000 images | CC BY-SA 3.0 | Image classification |
Text Extraction for OCR
- Size
- About 1,560 invoice images (JPG/PNG) paired with XML entity annotations, including extracted tabular data
- Licence
- CC0: Public Domain
About 1,560 pairs of invoice images in JPG or PNG with XML files holding extracted entities such as invoice number, date, business name, telephone and address, plus extracted tabular data. A ready-made document-NER set.
Developing invoice-specific entity extraction, improving recognition of tabular data in complex images, and building OCR robust to typographic and quality errors.
Source: innovatiana.com
OCRFlux Bench Single
- Size
- 2,000 annotated pages (Markdown), bilingual EN/ZH
- Licence
- Apache 2.0
2,000 PDF pages with manual Markdown annotations, bilingual across English and Chinese, for evaluating how accurately OCR systems parse complex layouts including titles, tables and structured text into Markdown output.
Measuring OCR accuracy on complex layouts, testing multilingual OCR on English and Chinese, and improving Markdown output from scanned or PDF images.
Source: innovatiana.com
OCR Benchmark
- Size
- 1,000 examples, about 386 MB, Parquet and JSON format
- Licence
- MIT
A 1,000-example benchmark of about 386 MB in Parquet and JSON, for comparing how well multimodal LLMs perform OCR and extract structured JSON from documents. Evaluation-sized rather than training-sized.
Measuring OCR accuracy of multimodal models, comparing structured JSON extraction quality across LLMs, and improving joint visual and text comprehension.
Source: innovatiana.com
BLIP3 OCR 200M
- Size
- About 96 million rows in Parquet (image URLs with OCR text and boxes); the BLIP-3 paper describes 200 million images
- Licence
- Apache 2.0, but released for research purposes only; records hold image URLs and OCR metadata, not the images
Roughly 96 million Parquet rows of OCR annotations for web images that the records reference by URL rather than include, built to help vision-language models read text-heavy visuals such as documents, diagrams and screenshots. Its Hugging Face release shows about 96 million rows, against the 200 million images the BLIP-3 paper describes (the card itself says about 2 million samples), so check what a row holds before quoting a size. Salesforce releases it for research purposes only.
Pre-training VLMs on text-heavy images, raising OCR accuracy on real-world documents, and boosting multimodal reasoning.
Source: innovatiana.com
TextOCR: Extraction of text on natural images
- Size
- 28,134 images, about 900,000 word annotations, JPEG and JSON
- Licence
- Annotations CC BY 4.0; images CC BY 2.0 (Open Images, via TextVQA)
28,134 natural-scene images from TextVQA with about 900,000 word-level polygon annotations in JSON. Polygon rather than box annotation lets it cover curved and partially occluded text. The annotations are CC BY 4.0 and the Open Images photos CC BY 2.0, so attribution is required.
Training OCR on complex text such as curved or partially visible words, improving VQA or multimodal captioning, and testing robustness across fonts and backgrounds.
Source: innovatiana.com
Sujet Finance QA Vision 100k
- Size
- 107,050 QA pairs, 9,801 images, structured format (JSON/image)
- Licence
- Apache-2.0
107,050 English question-answer pairs generated from 9,801 images of financial documents such as reports, tables and invoices, in a structured JSON and image format. Professional document VQA rather than natural scenes.
Developing visual question answering for complex documents, improving financial document analysis via OCR and semantic understanding, and refining multimodal LLMs.
Source: innovatiana.com
MNIST
- Size
- 70,000 images (60,000 for training, 10,000 for testing), PNG or IDX format
- Licence
- Creative Commons Attribution-Share Alike 3.0 (CC BY-SA 3.0)
70,000 centred greyscale 28x28 images of handwritten digits labelled zero to nine, split 60,000 train and 10,000 test, in PNG or IDX.
Training image classification, benchmarking CNNs and autoencoders, demonstrating supervised pipelines, and experimenting with dimensionality reduction or GANs.
Source: innovatiana.com
How this list was put together
Found by searching Dataset Finder for datasets whose documentation describes the task, then checked by hand. A keyword match is not a fit — the first pass picked up a coral survey for the depth-estimation list because it was recorded below fifty metres — so every entry was read against what this page covers, and the ones that matched a word without matching the task were removed.
Sizes and licences come from each dataset's own documentation, vague wording included. Where we found a re-upload or catalogue card stating a licence that the original release contradicts — a CC0 tag on Flickr photographs, MIT on data whose terms require a signed agreement — the entry gives the original terms and says so. The comparison table abbreviates licences; hover a cell for the full wording.