Physical AI data has grown enormously in three years, and a striking amount of the biggest and best of it cannot be used commercially. That constraint shapes the field more than the scale numbers do.
What the term means, and who is pushing it
The most citable primary definition comes from NVIDIA's Cosmos paper: “Physical AI is an AI system equipped with sensors and actuators” — “the sensors allow it to observe the world, and the actuators allow it to interact with and modify the world.”
It is worth being clear that this is largely a vendor frame. NVIDIA is the principal driver of the term, which anchors Cosmos, its simulation stack, its humanoid foundation models and its PhysicalAI-* dataset family as one platform story.
The terms the research literature actually uses are embodied AI, world model and vision-language-action model. Others working in the area — Meta, Google DeepMind, Physical Intelligence, AgiBot, 1X, Figure — mostly prefer those. If you are searching for papers rather than product pages, search for those instead.
The open corpora, with licences
Scale figures are as published by each source. The licence column is the one to read first, because it disqualifies more candidates here than in any other area of machine learning.
| Dataset | Published scale | Licence |
|---|---|---|
| AgiBot World Colosseo | 1,001,552 trajectories, 2,976.4 hours, 217 tasks, 106 scenes | CC BY-NC-SA 4.0 — non-commercial |
| Open X-Embodiment | 1M+ trajectories, 22 embodiments, 60 pooled datasets, 527 skills | aggregate packaging CC-BY; per-dataset licences not stated |
| DROID | 76k trajectories, 350 hours, 564 scenes, 86 tasks, 50 collectors | CC BY 4.0 |
| BridgeData V2 | 60,096 trajectories across 24 environments | CC BY 4.0, stated on the project page |
| Ego4D | 3,670 hours, 931 camera wearers, 9 countries | bespoke Ego4D agreement — signed data-use agreement required |
| Ego-Exo4D | 1,286 hours ego+exo, 740 participants, 13 cities | not stated in the paper |
| EgoDex | 829 hours, 90M frames, 338,000 demos, 194 tasks | CC BY-NC-ND — non-commercial, no derivatives |
The data pyramid
Current practice stacks three tiers, and GR00T N1 publishes its own totals in a way that makes the proportions concrete.
Base: human video. Egocentric footage of people doing things. GR00T N1's human-video layer totals 181.3M frames across 2,517.0 hours, of which Ego4D supplies 2,144.7 hours. Cheap per hour, abundant, but there are no robot actions in it — the model learns what tasks look like, not how to actuate.
Middle: synthetic data. Simulation rollouts plus neural-generated trajectories. In GR00T N1 this tier is synthetic only.
Top: real robot data. In GR00T N1 this is the team's own GR-1 teleoperation together with pooled cross-embodiment sets such as Open X-Embodiment and AgiBot-Alpha. Only the in-domain part is actually your robot in your environment, and it is the only layer that cannot be scraped or downloaded. GR00T N1's own teleoperation contribution is just 88.4 hours against a grand total of 8,375.7 — about 1%.
That 1% is the expensive part, and the scaling question everyone wants answered is how much of it they need.
How much in-domain data is enough
There is one solid published answer, and it is more useful than the raw scale numbers. Data Scaling Laws in Imitation Learning (ICLR 2025) finds that generalisation “scales approximately as a power law with the number of training objects, training environments, or training environment-object pairs” — and, critically, that “the diversity of environments and objects is far more important than the absolute number of demonstrations.”
With the setting held fixed, there is no clear power-law relationship between demonstration count and performance: performance plateaus at 400, 800 and 1,600 demonstrations for 8, 16 and 32 environment-object pairs respectively.
Its recommendation, verbatim: collect “in as many diverse environments as possible, with only one unique object in each environment”, at “50 demonstrations per environment-object pair”, and “when the total number of environment-object pairs reaches 32, it is generally sufficient to train a policy” for novel environments and unseen objects. That is roughly 1,600 demonstrations, and the study reports approximately 90% success in novel environments with unseen objects.
One caveat the paper states itself: for one harder task its model predicts 1,191 environment-object pairs would be needed to reach a near-perfect score, and it leaves that unverified. The 32-pair figure is a finding for the tasks studied, not a universal constant.
Where to start
If you are training a manipulation policy, the conventional path is to pretrain on pooled cross-embodiment data and fine-tune on your own. OpenVLA is the clearest reference point: a 7B model trained on 970k real robot demonstrations drawn from a pool of over 70 datasets, which outperforms “closed models such as RT-2-X (55B) by 16.5% in absolute task success rate across 29 tasks” with seven times fewer parameters — at a cost of 21,500 A100-hours.
For commercial work, filter by licence before anything else. DROID is CC BY 4.0 and well documented. Open X-Embodiment's aggregate packaging is CC-BY but its repo states no per-dataset licences, so if you need certainty you must trace the constituent datasets individually — a real piece of work, and better done before training than after.
And expect to collect your own top tier regardless. No open dataset contains your robot in your environment, and that layer is what determines whether the policy works where you need it to.
Questions people ask
Can I use AgiBot World commercially if I only pretrain on it?
CC BY-NC-SA prohibits commercial use and adds share-alike, and whether model weights trained on such data constitute a derivative work is legally unsettled rather than clearly resolved. This is a question for your lawyers, not an inference to make from the licence text — and it is the reason several commercial labs train on DROID and their own data instead.
Is simulation data a substitute for real data?
Partly, and it is improving. Simulation gives you volume, perfect labels and dangerous scenarios safely, but policies trained purely in simulation degrade on transfer. Current practice uses simulation for scale and real data for grounding, with domain randomisation to narrow the gap.
What format does this data come in?
Two main ones. Open X-Embodiment uses RLDS, serialised into TFRecord files. The community standard on Hugging Face is LeRobotDataset — Parquet for the low-dimensional signals plus MP4 per camera. Conversions between them exist, which is why LeRobot ports of OXE are common.
Do I need a humanoid to use humanoid data?
No, and cross-embodiment transfer is the central bet of the field. Open X-Embodiment converts actions into a shared 7-dimensional shape so that data from 22 different embodiments can train one model, though coordinate frames and action conventions are not aligned across its datasets. Transfer is imperfect but demonstrably better than same-embodiment-only training.