Robot navigation data: environments rather than demonstrations

Published 1 September 2026 · 5 min read

Manipulation data is demonstrations. Navigation data is mostly environments — and that difference changes where the data comes from, what it costs, and which licence you need to read.

The shape of the problem is different

A manipulation dataset is a set of trajectories: here is the task, here is what the operator did. You collect more by doing the task more times.

Navigation is largely the reverse. The behaviour — move to a goal without colliding — is simple to specify and often generated by a planner rather than demonstrated by a human. What is scarce is places: varied, realistic, traversable environments with enough geometric and visual diversity that a policy generalises to a building it has not seen.

So navigation research is dominated by simulators with scene assets, and by the question of where those scenes come from. The two sources are 3D scans of real buildings and synthesised or artist-built interiors, and they trade off differently: scans carry real clutter and real sensor artefacts but come with reconstruction holes; synthesised scenes are clean and controllable and can look it.

Two top-down renders of a simulated living room with leather sofas, an armchair, a rug and a coffee table. In the left version, red crosses mark paths and objects an agent cannot reach between the furniture; in the right version the furniture is rearranged, green arrows mark open paths and green ticks mark objects that can now be reached. A legend reads "Inaccessible Path or Object(s)" and "New accessible path, can access all Objects". — Two versions of one simulated living room from AI2-THOR's release notes. Rearranging the furniture turned unreachable objects into reachable ones: for a navigation agent the layout of the place decides which episodes can exist at all. Image: allenai/ai2thor · Apache-2.0 repository

What you need beyond the scenes

Three further ingredients, each with its own supply problem.

Episode definitions. Start and goal pairs with a known navigable path, so success is measurable and difficulty is controlled. These are generated from scene geometry rather than collected, which is convenient — and means their difficulty distribution is a design choice you should check.
Semantic annotation, if the task is more than point-to-point. “Go to the kitchen” or “find a mug” requires object and region labels in the scene, which is annotation effort on top of reconstruction effort.
Real-world validation. A simulated success rate is close to meaningless as a transfer estimate. Whatever you train in simulation, the evaluation that matters happens in a real building, and that is a hardware and logistics cost rather than a data one.

Four rows of three panels each, labelled rgb, semantic and depth: photorealistic indoor renders of a window ledge and an ornate hall, their matching flat-colour semantic segmentation maps, and their viridis-coloured depth maps, with the word 'stop' printed between episodes. — What a navigation agent receives at each simulation step: RGB, semantic labels and depth, perfectly synchronised. Simulation hands you clean semantic and depth channels that no real deployment will have. Image: facebookresearch/habitat-lab · MIT repository

The licence question sits under the simulator

This is the practical trap in navigation work, and it is a specific instance of the pattern that runs through all of Physical AI: permissive code, constrained data.

The simulator is software, and the mainstream ones are generally open. The scene assets are data, with separate terms, and they typically arrive as a default download that the getting-started guide tells you to fetch — which makes it very easy to train on them without ever having seen a licence.

The simulators themselves are generally permissive — Habitat-Lab and Habitat-Sim are MIT — but the scene datasets used with them carry separate terms, some of them non-commercial.

The procedure is straightforward: identify every scene asset pack and object library in your pipeline, find each one's licence, and record what you found, before you train anything you intend to ship.

Where real-world navigation data comes in

Simulation dominates, but real data has two roles that simulation cannot fill.

Validation, which is the non-negotiable one. The appearance and dynamics gaps described in our sim-to-real post apply fully here: rendered corridors differ from camera footage in ways that matter, and real buildings contain glass, reflective floors, moving people and clutter that scene assets under-represent.

Fine-tuning on the target environment. If you are deploying into a known building, collecting real traversals there is the highest-value data you can get — the same logic as in-domain teleoperation for manipulation, and the same conclusion from the published scaling work: environment diversity is what generalisation scales with, so a handful of real buildings is worth a great deal.

Egocentric human video is also relevant here, and more directly than for manipulation. People walking through buildings produce exactly the visual experience a navigating agent needs, and the corpora described in our egocentric video post contain a great deal of it — subject to the same licence caveats.

Three rows of seven grey occupancy-grid maps of corridors and rooms. Each map has a green trajectory from a red start dot to a white end marker and a red rectangle marking the target object. — Paths (green) taken by a real robot in a university building while searching for a target object (red box), on maps built as it moved: a learned policy (top row), a classical pipeline (middle) and a hybrid of the two (bottom). The learned skills were trained in simulation; on the real robot the pure learned policy missed its target twice. Image: Figure 8 of the article · CC BY 4.0 · from Staroverov et al. 2023, Robotics, “Skill Fusion in Hybrid Robotic Framework for Visual Object Goal Navigation”, doi:10.3390/robotics12040104

A practical sequence

If you are starting a navigation project, the order that wastes least:

Resolve the licence question first, across the simulator, the scene assets and any object libraries. It is much cheaper now than after training.

Build a real-world evaluation route before you train anything, in the actual kind of building you are deploying into. Measure the baseline, whatever it is.

Train in simulation with aggressive domain randomisation across appearance and layout, then fine-tune on real traversals from the target environment, and evaluate on the real route every time. Simulated metrics are for iteration speed, not for decisions.

Domain randomisation does not make the simulator realistic. It makes it wide, so the real world becomes one more sample the policy has already seen — which is why randomising the wrong parameters buys nothing.

Questions people ask

Why is navigation so much more simulation-heavy than manipulation?

Because the behaviour is easier to generate and the environments are harder to obtain. You can produce millions of navigation episodes from one scene with a planner, whereas a manipulation demonstration needs a human and a rig. The bottleneck moves from behaviour to places.

Do I need semantic labels in my scenes?

Only if your task is semantic. Point-goal navigation needs geometry and traversability. Object-goal or instruction-following navigation needs object and region labels, which is substantially more annotation and narrows your choice of scene collections considerably.

Can I use building scans I make myself?

Yes, and for a known deployment environment it is often the best option — you get the actual target geometry and clean licence terms. Consumer depth sensors and photogrammetry have made this far more accessible, and a handful of real target buildings is valuable given that environment diversity is what generalisation scales with.

Is navigation data useful for manipulation, or vice versa?

Only loosely. They share visual representation learning, which is why general video pretraining helps both. The action spaces are unrelated — base velocity versus end-effector pose — so the action-grounding layers do not transfer. Mobile manipulation needs both and is harder than either.

Keep reading

Physical AIRobot manipulation datasets, ranked by what you can do with them Physical AIEmbodied AI datasets beyond manipulation Physical AIPhysical AI datasets: what exists, and what you may actually use