The LeRobotDataset format, and what changed in v3

What it standardises

The LeRobot v3 docs describe it as “a standardized format for robot learning data” — one interface over sensorimotor signals and multi-camera video, plus the collection metadata that makes a recording interpretable: task description, robot type, frame rates.

Robot data is awkward precisely because it is heterogeneous. A single episode is a high-frequency stream of joint states and actions, several synchronised video feeds at a different rate, and a natural-language task label. Storing that well means storing each part in the format suited to it, which is what the three pillars do.

The three pillars

The docs state the division explicitly.

• Tabular — “low-dimensional, high-frequency signals (states, actions, timestamps) stored in Apache Parquet.” Columnar, so you can scan actions across a whole dataset without touching the video.
• Visual — “camera frames concatenated and encoded into MP4. Frames from the same episode are grouped; videos are sharded per camera for practical sizes.” Video codecs rather than per-frame images, which is a very large storage difference.
• Metadata — “JSON/Parquet records describing schema (feature names, dtypes, shapes), frame rates, normalization stats, and episode segmentation (start/end offsets into shared Parquet/MP4 files).”

meta/info.json        canonical schema (features, shapes/dtypes), FPS, codebase version,
                      and path templates to locate data/video shards
meta/stats.json       global feature statistics (mean/std/min/max) for normalization
                      -> exposed as dataset.meta.stats
meta/tasks.parquet    natural-language task descriptions mapped to integer IDs
meta/episodes/        per-episode records (lengths, tasks, offsets) as chunked Parquet
data/                 frame-by-frame Parquet shards; each file holds many episodes
videos/               MP4 shards per camera; each file holds many episodes

Why v3 abandoned one-file-per-episode

In v2, each episode was its own file and episode boundaries were recovered from filenames. That is the obvious design and it works well up to a point.

The limit is file count. A million small files makes dataset initialisation slow, strains object stores and filesystems, and turns listing the dataset into a significant operation. The docs state the motivation directly: “to scale to millions of episodes… fewer, larger files ⇒ faster initialization and fewer issues at scale.”

So v3 concatenates many episodes into each Parquet and MP4 shard and resolves boundaries through “relational metadata: episode boundaries and lookups are resolved through metadata, not filenames.” The stated design principle is “decoupling storage from the user API” — the Python interface still hands you episodes while the layout underneath is free to change.

This is the same lesson the Parquet ecosystem learned about small files, arriving in robotics.

The feature keys

The docs show sample keys — observation.state, action, observation.images.<camera>, timestamp — presented as an example rather than as a required schema.

That distinction matters. The format does not mandate a fixed set of features; it mandates that whatever features you have are declared in meta/info.json with their shapes and dtypes. The dotted naming convention is a convention, and the loader reads your declaration.

Worth knowing what the docs do not publish: chunking and file-size defaults, the required info.json field list, the column names inside meta/episodes/, and codec details. If you are writing a converter rather than using the library's own writer, read the library source rather than relying on the documentation page.

What consumes it

The strongest argument for the format is the breadth of what now reads it. LeRobot implements policies in pure PyTorch, all consuming LeRobotDataset, across imitation learning (ACT, Diffusion, VQ-BeT), reinforcement learning (HIL-SERL, TD-MPC) and vision-language-action models (the Pi0 family, SmolVLA and others).

One on-disk layout feeding imitation learning, RL and VLAs is precisely why the v3 scaling change mattered — the format outgrew its original use. Evaluation runs through lerobot-eval against standard benchmarks including LIBERO and MetaWorld.

The library is Apache 2.0. Datasets published in the format carry their own per-repository licences, and the format implies nothing whatsoever about them — a point worth internalising given how many large robotics datasets are non-commercial.

Questions people ask