Robotics data has grown enormously, and most of the sets below carry permissive licences. The catch is in the 3D scene data underneath, where the terms of the original scans travel with everything built from them.
This category spans two quite different needs. Robot learning wants trajectories: synchronised observations and actions from a robot performing a task, which is the data you cannot download for your own platform and must collect. 3D perception wants geometry: point clouds, meshes and depth, for reconstruction, grasping and scene understanding.
The licensing pattern here is worth stating plainly, because it recurs. A permissive licence on a robotics or 3D dataset does not always clear the data inside it: scene sets are often built from scans released under their own terms, and those terms still apply. ScanNet, below, requires a signed agreement for its data although its code is MIT-licensed, and SceneSplat-7K is listed as CC BY-SA 4.0 while being built from ScanNet and Matterport3D. Check the data licence separately from the tooling licence, and the source scans separately from the derived set, before you train rather than after.
| Dataset | Modality | Size | Licence | Best for |
|---|---|---|---|---|
| Jack of All Trades (JAT) Dataset | Multimodal | 258M examples | Apache 2.0 | Understand and produce text and image |
| PhysicalAI Robotics GraspGen | 3D | 57M poses | CC BY 4.0 | AI models for robotic gripping |
| ALMI-X Whole-Body Motion Dataset | Multimodal | Tens of thousands of trajectories | Apache 2.0 | Robotic agents |
| VIKI-R | Multimodal | 22,637 examples | Apache 2.0 | Robot selection and task planning |
| SO101 Object in Box v0.4 Fixed | Multimodal | 101 episodes | Apache 2.0 | Object manipulation algorithms in robotics |
| SVLA SO100 Sorting Dataset | Video | 18 videos | Apache 2.0 | Vision-language-action models in robotics |
| Robo2vlm Reasoning | Multimodal | 5,150 examples | Apache 2.0 | Assessing visual reasoning of vision-language models |
| NYU Depth V2 | Multimodal | 47,584 images | Unclear — see below | Monocular depth estimation for indoor scenes |
| ScanNet | Image | 1,513 scans | Gated access | 3D semantic segmentation |
| Ego4D Video - Embodied planning dataset | Multimodal | Hundreds of hours | Gated access | Vision-language models to follow instructions |
| SceneSplat-7K | Image | 7,916 scenes | Research only | NeRF or 3D Gaussian Splatting models |
Jack of All Trades (JAT) Dataset
- Size
- Over 258 million examples, 1.09 TB, Parquet formats (text, images, RL episodes, captions)
- Licence
- Apache 2.0
Over 258 million examples across 1.07 TB of Parquet, combining text, images, reinforcement-learning demonstrations and image-caption pairs in one corpus for training generalist multimodal agents.
Training agents that understand and produce text and image content, leveraging RL demonstrations to learn complex behaviours, and developing multi-task architectures.
Source: innovatiana.com
PhysicalAI Robotics GraspGen
- Size
- 57 million grasp poses in WebDataset format (.tar), JSON for splits
- Licence
- CC-BY 4.0
57 million simulated grasp poses in WebDataset .tar archives with JSON split files, generated across 8,515 Objaverse XL 3D objects and three gripper types. Published by NVIDIA. It holds grasp poses and object references, not images.
Training AI models for robotic gripping in simulation, testing Sim2Real approaches for 3D object manipulation, and creating benchmarks for different gripper types.
Source: innovatiana.com
ALMI-X Whole-Body Motion Dataset
- Size
- Tens of thousands of trajectories (81,549 rows in the dataset viewer) in .npy and texts in .tar.gz (about 200 steps per episode)
- Licence
- Apache 2.0
Tens of thousands of simulated MuJoCo humanoid trajectories (the dataset viewer lists 81,549 rows) in .npy with text in .tar.gz, around 200 steps per episode, pairing lower-body and upper-body motion with automatically generated language descriptions.
Training robotic agents in complex locomotion and motion imitation, linking textual commands to physical actions, and testing simulated RL policies before deployment.
Source: innovatiana.com
VIKI-R
- Size
- 22,637 examples (scene images with text prompts), Parquet
- Licence
- Apache 2.0
22,637 examples from VIKI-Bench, the embodied multi-agent cooperation benchmark of the VIKI-R paper, whose levels are agent activation, task planning and trajectory perception. Each record pairs one scene image with a prompt describing the available robots and a ground-truth answer: images rather than video clips, and no rationale field.
Training robot selection and task planning for multi-robot cooperation, evaluating embodied multi-agent reasoning in vision-language models, and studying trajectory perception.
Source: innovatiana.com
SO101 Object in Box v0.4 Fixed
- Size
- 101 episodes, 22,260 frames at 30 fps: front-camera video (640×480, H.264) with six-joint action and state data, LeRobot format
- Licence
- Apache 2.0
22,260 frames in 101 episodes recording object handling inside a box: front-camera video at 640×480 with six-joint action and state readings, in the LeRobot format. Captured on the SO-101 arm, it suits manipulation learning in constrained workspaces.
Developing object manipulation algorithms in robotics, studying recognition and placement in constrained environments, and training vision and physical interaction models.
Source: innovatiana.com
SVLA SO100 Sorting Dataset
- Size
- 18 videos (480x640, 30fps, AV1 codec), 6,633 frames, action and observation data in parquet format
- Licence
- Apache 2.0
18 videos at 480x640 and 30 fps in AV1, totalling 6,633 frames, paired with action and observation data in Parquet from an SO-100 robot arm performing sorting. Part of the LeRobot collection.
Developing vision-language-action models in robotics, analysing robotic movement for sorting and handling, and testing multimodal learning on video plus sensor data.
Source: innovatiana.com
Robo2vlm Reasoning
- Size
- 5,150 multiple-choice examples (4,635 train, 515 test) with images, questions and generated reasoning
- Licence
- Apache 2.0
5,150 multiple-choice questions about images of robots handling objects, each with reasoning traces generated by Gemini 2.5 Pro to support the correct answer. Built to check whether vision-language models can justify answers about robotic scenes.
Assessing visual reasoning of vision-language models, strengthening understanding of robotic scenes in context, and measuring whether models back answers with consistent reasoning.
Source: innovatiana.com
NYU Depth V2
- Size
- 47,584 training and 654 validation RGB images with dense depth maps (FastDepth preprocessing), Parquet
- Licence
- No licence stated on the official NYU page (citation requested); this Hugging Face copy's card says MIT and its metadata tag Apache 2.0
RGB images paired with dense depth maps from NYU Depth V2's indoor Kinect recordings, in the FastDepth preprocessing: 47,584 for training and 654 for validation. This copy holds depth only; the segmentation labels of the original 1,449-image labelled set are not in it. The long-standing baseline for monocular depth estimation.
Training monocular depth estimation for indoor scenes, building perception for home robots or AR/VR glasses, and studying 3D scene reconstruction from RGB-D.
Source: innovatiana.com
ScanNet
- Size
- 1,513 annotated RGB-D scans, 2D/3D formats (images, voxels, meshes, depth maps)
- Licence
- ScanNet Terms of Use for the data (signed agreement required); code under MIT
1,513 annotated indoor RGB-D scans covering 20 object classes, delivered across 2D and 3D formats including images, voxels, meshes and depth maps. The reference dataset for indoor 3D semantic segmentation. Only the code is MIT: the data is released under the ScanNet Terms of Use, which you sign to request access.
3D semantic segmentation, interior scene reconstruction, object detection and tracking in enclosed spaces, and robotic perception and indoor navigation.
Source: innovatiana.com
Ego4D Video - Embodied planning dataset
- Size
- Hundreds of hours of egocentric videos + text instructions, video formats + JSON
- Licence
- Ego4D licence agreement required; this re-upload is labelled Apache 2.0
Hundreds of hours of first-person video derived from Ego4D, with automatically generated and then manually verified language instructions, in video formats plus JSON. Oriented towards embodied planning and multimodal reasoning. The footage is Ego4D's, which is licensed only through the Ego4D licence agreement; the Apache 2.0 tag on this re-upload does not replace it.
Training vision-language models to follow instructions in complex settings, evaluating multimodal reasoning through embodied planning, and building agents that act physically.
Source: innovatiana.com
SceneSplat-7K
- Size
- 7,916 scenes, 11.27 billion 3DGS points, `.npy`, `.json` files and metadata in `.csv`
- Licence
- CC BY-SA 4.0 covers only the code, scripts and metadata; the scenes come from ScanNet, Matterport3D and other indoor sets, which restrict use to non-commercial research and education
7,916 indoor 3D scenes reconstructed from sources including ScanNet and Matterport3D, carrying 11.27 billion Gaussian Splatting points with poses and annotations in .npy, .json and .csv. Built for 3DGS and NeRF work. Access is gated, and the CC BY-SA 4.0 licence covers only the code and metadata: every source dataset restricts the scenes to non-commercial research and education. The card's statistics after later data updates total 5,384 scenes.
Training NeRF or 3D Gaussian Splatting models for photorealistic rendering, evaluating indoor 3D reconstruction from RGB-D, and testing 3D vision-language architectures.
Source: innovatiana.com
How this list was put together
Drawn from the datasets in Dataset Finder filed under this topic, then checked by hand against what this page covers. A tag is a filing decision, not a description, so records that carried the tag without being about the topic were removed, and relevant records filed elsewhere were added. The order puts the closest fit first.
Sizes and licences come from each dataset's own documentation, vague wording included. Where we found a re-upload or catalogue card stating a licence that the original release contradicts — a CC0 tag on Flickr photographs, MIT on data whose terms require a signed agreement — the entry gives the original terms and says so. The comparison table abbreviates licences; hover a cell for the full wording.