FAQ

Physical AI

Teleoperation data collection: the layer you cannot download

What it costs per task, the hardware people actually use, and the published finding that should reshape your collection plan.

Every other layer of a robot training mix can be downloaded. In-domain teleoperation cannot, which makes it the expensive part of any serious robotics project — and the part where a published scaling result can save you most of the money.

Why this layer is irreducible

GR00T N1’s published totals make the structure of a modern training mix concrete, and the proportions are striking. Human video contributes 2,517.0 hours. Robot data contributes 3,288.8 hours, and simulated and neural-generated trajectories 2,569.9. The grand total is 592.9M frames across 8,375.7 hours.

Three rows of six first-person VR frames labelled make sandwich, iron cloth and pack basket. Gloved hands are visible in front of robot arms working at tables, and in the first row a floating camera window shows the robot's view.
What the operator sees inside OPEN TEACH's virtual-reality headset while recording demonstrations for three tasks: make a sandwich, iron cloth and pack a basket. The robot's camera feed streams into the headset at 90 Hz while the operator's tracked hands drive the robot. Every frame of this data costs a person's time, which is why this layer cannot be scraped. Image: Fig. 3 of the paper · CC BY 4.0 · from Iyer et al. 2024, “OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulation”, arXiv:2403.07870

Of that, the team's own teleoperation on their GR-1 platform is 88.4 hours, 6.4M frames at 20 Hz. Roughly one percent of the total.

That one percent is the only data that is their robot, in their environments, doing their tasks. Everything else is a proxy — useful, cheap, abundant, and not quite the thing. The whole economics of robot learning is about how small you can make that irreducible layer while still getting a policy that works.

GR00T N1's pre-training mix, drawn to the hours in its paper. The paper's pyramid puts human video at the base, simulated and neural trajectories in the middle and real-robot data at the top, but by hours the top layer is the largest, because Open X-Embodiment and AgiBot-Alpha are in it. What you cannot download is the red slice: the team's own teleoperation, about 1% of the total.

What it costs per task

The imitation-learning literature publishes per-task collection numbers, and they are consistent enough to plan with.

ALOHA records “50 demonstrations for each task, except for Thread Velcro which has 100”, each episode taking “8-14 seconds for the human operator to perform” — 400 to 700 steps at 50 Hz. The rig cost under $20,000, which the appendix breaks down as $18k plus optional cameras.

Mobile ALOHA uses “only 50 human demonstrations per task” (two tasks used 20) on a $32,000 system, co-trained with 825 static episodes.

DROID is the large-scale version: 50 collectors over 12 months producing 76k trajectories and 350 hours. AgiBot World is the industrial version: a purpose-built facility over 4,000 square metres, skilled operators, VR and motion-capture control, human verification, a million trajectories.

So the atomic unit is about 50 demonstrations per task at roughly ten seconds each. The robot time is minutes. The cost is the rig, the operator, and the per-task setup and reset.

EffortDemos per taskHardware costScale achieved
ALOHA50 (100 for the hardest)under $20k6 tasks
Mobile ALOHA50 (20 for two tasks)$32k7 tasks
BridgeData V2—~$4k (WidowX 250)60,096 trajectories
DROID—Franka + 3 ZED cameras76k over 12 months
AgiBot World—dual-arm humanoid fleet1,001,552 trajectories

The finding that should change your plan

Data Scaling Laws in Imitation Learning is the most actionable published result in robot data collection, and it contradicts the intuitive approach.

The intuitive plan is: pick your task, collect as many demonstrations as you can afford, train. The paper finds that with environment and object held fixed there is “no clear power-law relationship” between demonstration count and performance — it simply plateaus. For 8, 16 and 32 environment-object pairs, performance plateaus at 400, 800 and 1,600 demonstrations respectively.

What does scale as a power law is diversity: the number of training objects, environments, or environment-object pairs. “The diversity of environments and objects is far more important than the absolute number of demonstrations.”

Hence the recommendation: collect “in as many diverse environments as possible, with only one unique object in each environment”, at “50 demonstrations per environment-object pair”, and “when the total number of environment-object pairs reaches 32, it is generally sufficient” — about 1,600 demonstrations, reportedly yielding around 90% success in novel environments with unseen objects.

Three line charts of normalised score against number of demonstrations on a log axis. Left: Pour Water and Mouse Arrangement from 64 to 6,400 demonstrations, flattening after about 200. Middle and right: Pour Water and Mouse Arrangement for 8, 16 and 32 environment-object pairs, each curve rising and then flattening, the 32-pair curve highest at about 0.9.
Left: collecting more demonstrations of the same set-ups raises scores on two tasks up to a few hundred demonstrations, then barely at all. Right: with 8, 16 or 32 environment–object pairs, scores climb and then level off as demonstrations increase, at a higher ceiling the more pairs there are. Past a few hundred demonstrations, new environments and objects buy more than more of the same. Image: Figure 7 of the paper · CC BY 4.0 · from “Data Scaling Laws in Imitation Learning for Robotic Manipulation”, arXiv:2410.18647

Collect for re-use

Since this data is the expensive part, record it so that it outlives the model you are training now.

Use a standard format from the start — LeRobotDataset is the community default and is what the policy implementations consume, so recording into it avoids a conversion later. Record more than you think you need: all camera streams, full joint states, gripper state, timestamps, and the natural-language task description. Dropping a stream to save disk is a false economy when re-collection means re-running the whole campaign.

Log the things that are not sensor data too: which operator, which environment, which object instance, what the lighting was, what went wrong. The scaling result above means environment and object identity are first-class metadata, not incidentals — you cannot compute diversity over a dataset that did not record it.

And keep failures. AgiBot World deliberately includes failure-recovery episodes at about 1% of the dataset. A policy that has only ever seen success has no idea what to do when the grasp slips.

A screenshot of the DROID Dataset Visualizer, "showing random subset of 1k trajectories from DROID". A left sidebar lists filter checkboxes with counts: scene types such as industrial office (586), kitchen (252), office (82) and laundry room (3); object types such as marker (163) and cloth (72); task types such as put (237) and move (141). Beside it is a grid of robot-camera thumbnails from different rooms.
DROID's own browser can filter by scene, object and task only because collectors logged them. The counts are a finding in themselves: in this 1,000-trajectory sample, 586 are labelled industrial office and three laundry room. Image (cropped): droid-dataset/droid · DROID dataset, CC BY 4.0 · from “DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset”, arXiv:2403.12945

The cheaper alternatives, honestly assessed

Three routes reduce the teleoperation burden, each with real limits.

A multi-panel photograph strip. The left panels show a person's hands holding a handheld gripper fitted with a wrist camera, collecting demonstrations in ordinary kitchens and workshops. The right panels show robot arms of different makes reproducing the same manipulation tasks in similar scenes.
Demonstrations collected with a handheld gripper rather than a robot, then transferred to several different arms. Collect once, re-use across arms — one of the few ways found so far to cut the cost of teleoperation data. Image: real-stanford/universal_manipulation_interface · MIT repository · from “Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots”, arXiv:2402.10329

Human video. Egocentric footage is orders of magnitude cheaper per hour and is the base of the data pyramid. It contains no robot actions, so it teaches what tasks look like rather than how to actuate. EgoDex is the interesting recent development — 829 hours with paired 3D hand tracking captured at recording time, 338,000 demonstrations across 194 tasks — though it is CC BY-NC-ND, so non-commercial and no-derivatives.

Simulation. Unlimited volume, perfect labels, no hardware risk, and a transfer gap. Best used for scale around a real core rather than instead of one.

Pretrained representations. The most promising route. V-JEPA 2 post-trained an action-conditioned model on “less than 62 hours” of existing DROID video and then controlled Franka arms in two labs with no data collected in either. If that generalises, the in-domain requirement shrinks substantially.

Domain randomisation does not make the simulator realistic. It makes it wide, so the real world becomes one more sample the policy has already seen — which is why randomising the wrong parameters buys nothing.

Questions people ask

How many tasks should I collect before training?

The published guidance points at breadth: roughly 32 environment-object pairs at 50 demonstrations each. If your application is genuinely one task in one fixed cell, the plateau figures suggest a few hundred demonstrations is where returns flatten — and that fixed setting is also why such policies generalise poorly.

VR controllers or motion capture?

Both are used at scale: DROID teleoperates with a Meta Quest 2, AgiBot World uses VR control and whole-body motion capture. VR is cheaper and quicker to set up; motion capture gives richer whole-body data and suits humanoids. Bimanual puppeteering rigs like ALOHA's are a third route and are what the low-cost literature is built on.

Can I pay someone else to collect it?

Yes, and for a well-specified task set this is often faster than building the capability in-house — the rig, the operators and the per-task setup are the cost, and they are amortised across clients. The thing to specify carefully is diversity of environments and objects, since that is what the scaling result says you are actually buying.

Does operator skill matter?

It affects the data's consistency, and AgiBot World's use of skilled operators plus human verification suggests the industrial view. Inconsistent demonstrations of the same task teach a policy to be inconsistent, in the same way contradictory instruction examples do for language models.

Stop hunting. Start searching.

Tens of thousands of curated datasets, searchable in plain language — with the licence and size checked before you download anything.

Search Dataset Finder →