Sim-to-real: what simulation data buys and what it costs

What simulation genuinely gives you

Four things that real collection cannot, and they are substantial.

Volume at negligible marginal cost. Once the environment exists, a thousand more episodes costs compute rather than operator hours and scene resets.

Perfect labels. Exact object poses, full segmentation masks, depth, contact forces, and the ground truth of what actually happened — all free, and several of them impractical to obtain on real hardware at all.

Dangerous and rare scenarios. The collision, the drop, the failure mode you need the policy to handle and cannot afford to stage repeatedly on a real robot.

Reset for free. Scene reset is a large part of the real cost of a collection campaign, and in simulation it is a function call. This is the underrated one.

The gap, and why it persists

A policy trained in simulation and deployed on hardware typically degrades, and the reasons sort into three groups.

Appearance. Rendered images differ from camera images in noise, dynamic range, motion blur, lens distortion and lighting response. This is the most tractable of the three.

Dynamics. Contact, friction, deformation and compliance are approximated. Rigid-body simulators handle rigid objects well and cloth, cables, granular material and anything deformable considerably less well. If your task involves those, expect the gap to be large.

The things not modelled. Actuator backlash, sensor latency, thermal drift, a slightly loose fixture. These are rarely in the model and reliably present in reality.

Domain randomisation is the standard mitigation: vary textures, lighting, physical parameters and camera placement during training so the policy cannot depend on simulator-specific regularities. It narrows the appearance gap well and the dynamics gap less so.

How teams actually combine them

The pattern across the published work is simulation for scale around a real core, rather than either alone.

GR00T N1’s data pyramid is the clearest statement of it: web data and human video at the base, synthetic data (simulation plus neural-generated trajectories) in the middle, and real-robot data at the top — the team's own GR-1 teleoperation together with pooled cross-embodiment sets such as Open X-Embodiment and AgiBot-Alpha. Simulation sits in the middle tier — it contributes scale and coverage, and neither the grounding nor the specificity.

Mobile ALOHA’s co-training result is the same idea with real data in both roles and is worth knowing because the numbers are so uneven: 50 in-domain demonstrations co-trained with 825 static episodes moved Rinse Pan from 0% to 80% and Call Elevator from 0% to 95%, while two other tasks did not improve at all.

That unevenness is the honest summary of mixing data sources. It can be transformative for some tasks and do nothing for others, and which is which is not reliably predictable in advance — which is an argument for having an evaluation set before you start.

The licence trap in scene assets

This catches people, because the thing you install and the thing you train on are licensed separately.

A simulator is software and is usually permissively licensed. The scene assets — the scanned buildings, the object meshes, the furnished interiors that make the simulator useful — are data, with their own terms, and they often arrive as a default download without a prominent licence statement.

Habitat-Sim and Habitat-Lab, for example, are MIT-licensed; the scene datasets commonly used with them carry their own terms, some of them non-commercial.

The actionable advice: before you train a commercial policy, check the licence of every scene asset pack and object library you are using, separately from the simulator. Record what you checked. This is the same pattern as LeRobot, whose code is Apache 2.0 while datasets in its format carry whatever licence their contributor chose.

A reasonable default plan

If you are starting out, the sequence that wastes the least effort:

Start from a pretrained policy rather than from scratch, since someone else has already paid the cross-embodiment pretraining cost. Build your evaluation set on real hardware first, because a simulated evaluation of a sim-trained policy tells you almost nothing about transfer.

Use simulation to explore — task variations, scene diversity, failure cases — with aggressive domain randomisation. Then collect real in-domain demonstrations, prioritising environment and object diversity over count, per the published scaling guidance.

Fine-tune on the real data with the simulated data co-trained in, and measure on hardware. If co-training does not help for your task, that is a real finding and it does not mean you did it wrong — Mobile ALOHA found exactly that for two of seven tasks.

Questions people ask