Agents that operate software need a data shape almost nobody else does: an interface state, an instruction, and the action that follows. This is the newest and fastest-moving category here.
The data has three layers and the useful datasets provide more than one. A visual layer — screenshots, and increasingly video of a session. A structural layer — the accessibility tree or DOM, which tells a model what elements exist rather than making it infer them from pixels. And an action layer — the clicks, keystrokes and scrolls, grounded to specific elements.
Two things make this category harder than it looks. Interfaces change, so a dataset of screenshots ages into a record of software that no longer looks like that, and benchmark scores on stale interfaces overstate real capability. And evaluation is genuinely difficult: there are usually many valid action sequences to a goal, so comparing against a single recorded trace penalises correct behaviour. The better benchmarks check the end state instead, which is also what makes them suitable for verifiable-reward training.
| Dataset | Modality | Size | Licence | Best for |
|---|---|---|---|---|
| MultiUI Dataset | Multimodal | 7.3M examples | ODC-By | Multimodal understanding of web user interfaces |
| Webcode2m | Multimodal | 3,171,024 instances | CC BY 4.0 | Multimodal models to generate web code |
| WebSight: Synthetic HTML Websites & Screenshots | Multimodal | 2.7M examples | CC BY 4.0 | Generate HTML from an image |
| GUI-net-1m: Large-Scale Image Dataset | Image | 895,000 images | Apache 2.0 | Deep learning models |
| Mobile UI Design | Image | 7,846 images | Apache 2.0 | Object detection for mobile UI |
| WebClick - Multimodal benchmark for web browsing | Multimodal | 1,639 images | Apache 2.0 | UI understanding by multimodal models |
| UI-Vision GUI Benchmark Dataset | Image | Images + JSON annotations… | MIT | Detect and locate UI elements |
| xLam Function Calling 60k | Text | 60,000 examples | Research only | Call API functions from natural language |
| ShowUI Desktop | Image | 7,496 annotations | Not stated | Understand and locate items |
MultiUI Dataset
- Size
- 7.3 million multimodal examples (text, images, UI), structured format
- Licence
- Open Data Commons Attribution (ODC-BY)
7.3 million multimodal examples drawn from one million websites, combining text, images and UI layouts extracted from web pages in a structured format. One of the largest open corpora for interface understanding.
Training multimodal understanding of web user interfaces, improving text recognition and document analysis, and developing agents that interact with complex UIs.
Source: innovatiana.com
Webcode2m
- Size
- 3,171,024 instances, PNG images, HTML/CSS code in text, Parquet files (~1.1 TB)
- Licence
- CC BY 4.0
3,171,024 instances pairing real web design screenshots in PNG with their HTML and CSS source and layout data, about 1.1 TB of Parquet. Larger and drawn from real pages, unlike the synthetic WebSight.
Training multimodal models to generate web code, building front-end AI design tools, and evaluating how well MLLMs understand interfaces visually and textually.
Source: innovatiana.com
WebSight: Synthetic HTML Websites & Screenshots
- Size
- 2.7 million examples, 317 GB, Parquet files
- Licence
- CC-BY 4.0
2.7 million synthetic pairs of HTML and CSS code with matching rendered screenshots, 317 GB of Parquet, generated from LLM-produced design ideas. The HTML is synthetic, but v0.2 pages embed real images, and the card asks you to respect the source content's licences and to disclose use of the dataset when you release a model.
Training models to generate HTML from an image, testing visual code generation or design-system algorithms, and building accessibility or interface analysis tools.
Source: innovatiana.com
GUI-net-1m: Large-Scale Image Dataset
- Size
- Approximately 895,000 images divided into several compressed archives, requiring a script for extraction and fusion
- Licence
- Apache 2.0
Roughly 895,000 GUI screenshot images split across several compressed archives that need a provided script to merge and extract. Large-scale interface imagery, though the page is vague about annotation structure.
Training deep learning models for large-scale computer vision, testing robustness and generalisation of recognition algorithms, and handling very large image volumes.
Source: innovatiana.com
Mobile UI Design
- Size
- Approximately 7,846 annotated images, 1.16 GB, Parquet format
- Licence
- Apache 2.0
About 7,846 annotated mobile interface screenshots in 1.16 GB of Parquet, with bounding boxes marking the position and class of UI elements such as text, images and groups. Mobile-specific rather than desktop or web.
Training object detection for mobile UI, automatically recognising interface elements such as text and images, and improving AI-assisted UI design tools.
Source: innovatiana.com
WebClick - Multimodal benchmark for web browsing
- Size
- 1,639 PNG/JPEG images, text instructions, bounding box coordinates in JSON
- Licence
- Apache 2.0
1,639 annotated web screenshots in PNG and JPEG, each paired with a natural-language click instruction and bounding-box coordinates in JSON. A compact evaluation set for web-agent click grounding.
Evaluating UI understanding by multimodal models, testing accurate click localisation from natural language instructions, and benchmarking web browsing agents.
Source: innovatiana.com
UI-Vision GUI Benchmark Dataset
- Size
- Images + JSON annotations in several subsets (functional, spatial, layout)
- Licence
- MIT
A desktop GUI benchmark of annotated application screenshots with JSON annotations, split into functional, spatial and layout subsets. Published by ServiceNow under MIT; no item count is given on the page.
Training models to detect and locate UI elements, evaluating functional and spatial understanding of components, and developing agents that use desktop interfaces.
Source: innovatiana.com
xLam Function Calling 60k
- Size
- 60,000 JSON examples, structured format: request, tools, parameters, responses
- Licence
- Research purposes only, per the Hugging Face card, which shows CC BY 4.0
60,000 JSON examples pairing a natural-language query with verified function-calling API calls, structured as request, available tools, parameters and response, spanning 21 API categories. Salesforce-published and verified. It is released under CC BY 4.0 on its Hugging Face card, which limits the release to research purposes.
Training models to call API functions from natural language, evaluating parameter understanding and correct execution, and benchmarking function-calling agents.
Source: innovatiana.com
ShowUI Desktop
- Size
- 7,496 element annotations on about 100 desktop screenshots, 328 MB, Parquet format
- Licence
- No licence stated on the Hugging Face card
7,496 annotated UI elements on about 100 desktop screenshots from OmniAct, in Parquet at 328 MB, with GPT-4o-written queries about each element's appearance, spatial relationships and function. Aimed at grounding natural-language instructions to interface elements; the screenshots are few, so it is a grounding set rather than a screenshot corpus.
Training models to understand and locate items in graphical interfaces, improving UI recognition for automation, and human-computer interaction research.
Source: innovatiana.com
How this list was put together
Drawn from the datasets in Dataset Finder filed under this topic, then checked by hand against what this page covers. A tag is a filing decision, not a description, so records that carried the tag without being about the topic were removed, and relevant records filed elsewhere were added. The order puts the closest fit first.
Sizes and licences come from each dataset's own documentation, vague wording included. Where we found a re-upload or catalogue card stating a licence that the original release contradicts — a CC0 tag on Flickr photographs, MIT on data whose terms require a signed agreement — the entry gives the original terms and says so. The comparison table abbreviates licences; hover a cell for the full wording.