Video is the most expensive modality to store, label and train on, and the most likely to be distributed as a list of links rather than as files.
Three practical realities shape this category. First, scale is measured in hours rather than items, and an hour of video is enormous compared with an hour's worth of images — budget storage and decode throughput before you commit to a set. Second, many video datasets are distributed as URL lists pointing at hosted platforms, which means availability decays as source videos are removed, and two researchers downloading the same dataset a year apart do not get the same data.
Third, annotation is harder to specify than for images. A label can apply to a whole clip, to a time span within it, or to a tracked object across frames, and these are not interchangeable. Check which granularity a dataset provides against what your task needs — clip-level labels can train temporal localisation only weakly, and cannot measure it.
| Dataset | Modality | Size | Licence | Best for |
|---|---|---|---|---|
| OpenS2V-5M | Multimodal | 5M triples | CC BY 4.0 | Video generation |
| Text2Video - Pika 2.2 Human Preferences | Multimodal | 756,000 responses | Apache 2.0 | Objectively evaluating text-to-video models |
| WM-ABench | Video | 100,000 videos | Apache 2.0 | Spatial, temporal and material perception |
| 360Motion Dataset | Video | 72,000 clips | Apache 2.0 | 3D motion generation or animation models |
| LEMONADE - Egocentric Video Language Assessment | Video | 36,521 pairs | CC BY 4.0 | Video comprehension in multimodal language models |
| Carla Car Monocular Camera 60 FPS | Video | 36,000 images | MIT | Monocular SLAM and visual odometry |
| SVLA SO100 Sorting Dataset | Video | 18 videos | Apache 2.0 | Vision-language-action models in robotics |
| MVBench | Video | 4,000 examples | MIT | Temporal and dynamic understanding |
| CameraBench | Video | 1,071 videos | CC BY 4.0 (repo) | Vision-language models on video perception |
| Shoplifting Video Dataset | Video | 182 files | CC BY 4.0 | Detect suspicious behaviour in stores |
| HowTo100M | Multimodal | 136M videos | Research only | Multimodal text-video-audio models |
| OpenVid-1M | Video | 1M videos | Research only | Text-conditioned video generation |
OpenS2V-5M
- Size
- 5 million subject-text-video triples in 720p, JSON for metadata and annotations (masks, bboxes in RLE)
- Licence
- CC-BY 4.0
Five million subject, text and video triples at 720p, with JSON metadata covering resolution, frame rate and aesthetic scores plus masks and bounding boxes in RLE format. Built for subject-consistent video generation.
Training video generation conditioned on subjects or text descriptions, developing multi-view video synthesis, and experimenting with text-subject-video models.
Source: innovatiana.com
Text2Video - Pika 2.2 Human Preferences
- Size
- 756,000 responses (JSON), low resolution GIF videos, and links to HD videos
- Licence
- Apache 2.0
About 756,000 human pairwise judgments in JSON comparing videos generated by Pika 2.2 from text prompts, with low-resolution GIF previews bundled and links out to the HD versions. A large preference corpus for video.
Objectively evaluating text-to-video models, training preference or reward models for RLHF, and analysing current limitations of video diffusion models.
Source: innovatiana.com
WM-ABench
- Size
- Over 100,000 annotated video sequences with multiple choice questions from 6 physical simulators
- Licence
- Apache 2.0
Over 100,000 annotated video sequences with multiple-choice questions, generated from six different physical simulators, probing whether vision-language models grasp and forecast physics. Simulator provenance gives ground-truth dynamics.
Assessing spatial, temporal and material perception, predicting physical dynamics such as falls and collisions, and validating multimodal vision-language models.
Source: innovatiana.com
360Motion Dataset
- Size
- 72,000 rendered clips (36,000 at 480×720 and 36,000 at 378×672; 99 frames, 30 fps), 12 cameras per scene
- Licence
- Apache-2.0
72,000 Unreal Engine-rendered clips (99 frames at 30 fps; 36,000 at 480x720 and 36,000 at 378x672) of 50 animal entities moving along 121 3D trajectory templates in six scenes, each filmed from 12 surrounding cameras. Built for trajectory-controlled multi-entity video generation (3DTrajMaster).
Training 3D motion generation or animation models, building agents that reproduce movement in immersive environments, and optimising vertical video synthesis for mobile or VR.
Source: innovatiana.com
LEMONADE - Egocentric Video Language Assessment
- Size
- 36,521 question-answer pairs over 30 MP4 videos, CSV for the Q/A
- Licence
- CC BY 4.0
36,521 closed-ended question-answer pairs in CSV, built over just 30 MP4 kitchen egocentric video clips, probing behaviour, temporal reasoning and biomechanics. Question density per video is extremely high, which is the design intent.
Evaluating video comprehension in multimodal language models, testing temporal reasoning over long sequences, and analysing posture and movement from video.
Source: innovatiana.com
Carla Car Monocular Camera 60 FPS
- Size
- 36,000 time-stamped PNG image files (nanosecond timestamps), YAML configuration files, and text
- Licence
- MIT
36,000 nanosecond-timestamped PNG frames from a simulated front-mounted monocular RGB camera on a virtual car in the Carla urban simulator, captured at 60 FPS, with YAML configuration and text files. Built for SLAM and visual odometry.
Developing monocular SLAM and visual odometry, training segmentation for embedded cameras, and simulating autonomous vehicle perception.
Source: innovatiana.com
SVLA SO100 Sorting Dataset
- Size
- 18 videos (480x640, 30fps, AV1 codec), 6,633 frames, action and observation data in parquet format
- Licence
- Apache 2.0
18 videos at 480x640 and 30 fps in AV1, totalling 6,633 frames, paired with action and observation data in Parquet from an SO-100 robot arm performing sorting. Part of the LeRobot collection.
Developing vision-language-action models in robotics, analysing robotic movement for sorting and handling, and testing multimodal learning on video plus sensor data.
Source: innovatiana.com
MVBench
- Size
- 4,000 examples in Parquet format, 328 KB
- Licence
- MIT
4,000 multiple-choice video questions in a 328 KB Parquet file, built by converting static image tasks into dynamic video ones. It spans basic perception through to complex cognition, and bundles most of its source videos; 320 NTU RGB+D clips must be downloaded separately under that dataset's licence.
Evaluating temporal and dynamic understanding in video models, testing temporal reasoning on complex tasks, and automatically creating reliable video benchmarks.
Source: innovatiana.com
CameraBench
- Size
- Approximately 1,071 video clips, 87 KB in parquet format
- Licence
- CC BY 4.0 (GitHub repository licence; the catalogue card says MIT)
Over a thousand manually annotated video clips built to improve understanding of camera movements in video. Expert labels and natural-language captions describe the motion in each clip, so generative and discriminative multimodal vision models can be evaluated on how well they perceive camera work.
Evaluating vision-language models on video perception, analysing camera movements in video sequences, and fine-tuning multimodal models for video tasks.
Source: innovatiana.com
Shoplifting Video Dataset
- Size
- 182 files (.mp4 videos) — 640×480 resolution — 30 FPS — 2 classes
- Licence
- CC BY 4.0
182 MP4 clips at 640x480 and 30 FPS in two classes, showing normal behaviour and simulated shoplifting staged in a lab. Small and synthetic in the sense of being acted, so generalisation to real CCTV is unproven.
Training models to detect suspicious behaviour in stores, testing action recognition on video streams, and prototyping intelligent surveillance systems.
Source: innovatiana.com
HowTo100M
- Size
- About 136 million video clips with narration, from 1.22 million YouTube videos (about 134,000 hours)
- Licence
- Research use; the videos remain under YouTube's terms (no MIT licence found; the repository's Apache 2.0 covers code)
Roughly 136 million video clips paired with their automatically transcribed narration, drawn from 1.22 million YouTube instructional videos: about 134,000 hours (15 years) of video across more than 23,000 tasks. Covers cooking, DIY, beauty and sport; individual videos remain under YouTube's terms.
Training multimodal text-video-audio models, pre-training for video search and captioning, and improving video-guided assistants and zero-shot video-text search.
Source: innovatiana.com
OpenVid-1M
- Size
- Over 1 million text-to-video videos in HD video format (≥512×512), including an OpenVidHD subset of 433K 1080p videos. Large ZIP files to decompress, CSV files for text-video pairs. Total size approximately several terabytes (OpenVidHD ~4.5 TB).
- Licence
- CC BY 4.0 tag, but the card says research and non-commercial purposes only; the source videos keep the licences of Panda-70M, ChronoMagic, Open-Sora-Plan and CelebV-HQ
Over a million HD text-to-video clips at 512x512 or above, with a 433,000-clip 1080p OpenVidHD subset, distributed as large ZIPs with CSV caption pairings and running to several terabytes. The card limits it to research and non-commercial purposes, and the source videos keep the licences of the datasets they came from.
Training text-conditioned video generation, improving video quality and resolution through fine-tuning, and researching video architectures on realistic data.
Source: innovatiana.com
How this list was put together
Drawn from the datasets in Dataset Finder filed under this topic, then checked by hand against what this page covers. A tag is a filing decision, not a description, so records that carried the tag without being about the topic were removed, and relevant records filed elsewhere were added. The order puts the closest fit first.
Sizes and licences come from each dataset's own documentation, vague wording included. Where we found a re-upload or catalogue card stating a licence that the original release contradicts — a CC0 tag on Flickr photographs, MIT on data whose terms require a signed agreement — the entry gives the original terms and says so. The comparison table abbreviates licences; hover a cell for the full wording.