First-person video is the cheapest large-scale data for teaching a model what human activity looks like. The three major corpora differ most in what is paired with the footage, and that pairing is what decides their usefulness.
The three corpora
Ego4D — unscripted daily life. “3,670 hours of daily-life activity video” from 931 unique camera wearers across 9 countries and 74 locations, assembled by 14 institutions. Its distinguishing feature is narration density: 3.85M sentences at 13.2 sentences per minute of video, with 1,772 unique verbs and 4,336 unique nouns, representing over 250,000 hours of annotator effort. Five benchmark families: episodic memory, hands and objects, audio-visual diarisation, social interactions, forecasting.
Ego-Exo4D — skilled activity, from both viewpoints at once. “1,286 hours of video combined”, 740 participants from 13 cities, 123 scene contexts, 5,035 takes of one to 42 minutes, 43 tasks and 689 keysteps across eight domains. Egocentric Aria capture synchronised with four to five tripod-mounted exocentric cameras, and three paired language corpora: first-person narration, third-person play-by-play, and expert commentary. Over 200,000 hours of annotator effort.
EgoDex — the scale jump, and the most action-adjacent. Apple Vision Pro footage with 3D hand and finger tracking captured at recording time: “829 hours of 30 FPS video and paired skeletal data with a total of 90 million frames”, “338000 task demonstrations across 194 tabletop manipulation tasks”, with upper-body joints at 30 Hz and 25 joints per hand.
| Ego4D | Ego-Exo4D | EgoDex | |
|---|---|---|---|
| Hours | 3,670 | 1,286 (ego+exo) | 829 |
| Participants | 931 wearers | 740 | not stated |
| Paired with video | dense narrations | 3 language corpora + exo views | 3D hand and finger tracking |
| Activity type | unscripted daily life | skilled procedural and physical | tabletop manipulation |
| Licence | bespoke agreement; signature required; published file named as a draft | not stated in the paper | CC BY-NC-ND |
The pairing is what matters
Raw first-person video is abundant. What makes a corpus valuable is what accompanies it, and the three have made three different bets.
Ego4D bet on language density. Thirteen narration sentences per minute makes it the natural corpus for grounding language in activity, and for the memory and forecasting tasks its benchmarks target.
Ego-Exo4D bet on viewpoint pairing. Having the same action from the wearer's view and from fixed cameras simultaneously is what lets a model learn the correspondence between the two — relevant for anyone who wants to learn from first-person footage and deploy on a robot that sees itself from outside. The expert-commentary track also makes it unusually suited to proficiency estimation.
EgoDex bet on 3D hand pose, and for robotics that is the most directly useful. A hand trajectory in three dimensions is much closer to an action than a pixel sequence is. It is also the restrictively licensed one, which is a recurring pattern in this field.
What models actually use
GR00T N1 publishes its human-video composition, which shows how these weight in practice. The total is 181.3M frames across 2,517.0 hours, made up of Ego4D 2,144.7 hours, HoloAssist 169.6, Ego-Exo4D 123.0, EPIC-KITCHENS 31.7, Assembly-101 19.3, RH20T-Human 16.3 and HOI4D 12.4.
Ego4D is 85% of it, which concentrates the licence question on a corpus whose terms are set by a bespoke, signature-only agreement.
EgoDex’s own comparison is the better sizing guide for manipulation specifically: Ego4D's human-object interaction subset gives 89k trajectories and 21M frames; DROID gives 76k trajectories, 86 tasks and 19M frames; EgoDex gives 338k trajectories, 194 tasks and 90M frames. By trajectory count EgoDex is four times either of them.
The licence situation, stated plainly
This is the part to resolve before anything else, and the honest summary is that it is unresolved in the primary sources.
Ego4D’s paper says only that the data is “publicly available under an Ego4D license” and does not state its terms. It is a bespoke agreement requiring a signature, not a Creative Commons licence, and the published document carries “Draft” in its filename. We have seen secondary summaries claiming it permits certain commercial uses while the footage remains protected and non-redistributable — that is a summary, not the agreement; the published agreement text itself does list “commercial or noncommercial product development” among permitted uses, but confirm the version you sign.
Ego-Exo4D’s paper states no dataset licence at all. The licence shown on the preprint is arXiv's non-exclusive licence to distribute the paper, not a grant for the data.
EgoDex is explicit and restrictive: “the dataset is publicly available under a CC-by-NC-ND license” — non-commercial, no derivatives.
So all three require legal review before commercial training, and for two of them the terms are not in the paper at all. If you are building a product, budget for that review early rather than discovering it after a training run.
What it can and cannot teach
Worth stating plainly, because the data pyramid framing can oversell the base layer.
It teaches visual representations of objects, scenes and activities; the structure and ordering of multi-step tasks; what human hands do and, with the pose-tracking corpora, where they are in space; and language grounded in physical activity.
It does not teach actuation. There are no joint states, no gripper commands, no action vectors. A model trained only on this knows what a task looks like and cannot execute one.
The most encouraging result for closing that distance is V-JEPA 2: pretrained on over a million hours of internet video, then action-conditioned on under 62 hours of existing robot footage, and able to control Franka arms in labs where no data was collected. If that pattern holds, large passive video plus a very small amount of robot data may be the efficient route — which would make these corpora more important, and their licensing more pressing.
Questions people ask
Which one should I use for robot learning?
EgoDex is the most directly useful because of the 3D hand tracking, if non-commercial use fits your situation. Ego4D is what the large published mixes actually weight most heavily. For commercial work, both need legal review, and robot-video corpora like DROID with clear CC BY 4.0 terms are the safer foundation.
Is there enough egocentric video to pretrain a world model?
Not from these corpora alone at the scale the frontier programmes use. V-JEPA 2 trained on over a million hours of internet video, against Ego4D's 3,670. These are curated, annotated corpora for learning specific things, not web-scale video pools.
Why is hand tracking such a big deal?
Because it is nearly an action label. A 3D trajectory of 25 joints per hand can be retargeted onto a robot hand or gripper in a way that a pixel sequence cannot, which makes pose-paired video the closest thing to free demonstration data — and is why EgoDex's licence is frustrating.
Can I record my own egocentric data?
Yes, and consumer headsets and smart glasses have made it far more practical — EgoDex is Vision Pro footage. If your tasks are specific, recording a few hundred hours yourself gives you data in your domain with licence terms you control, which solves both problems at once.