Rethinking Video Generation Model for the Embodied World

While video generation holds promise for embodied intelligence, current video models struggle with physical realism, and progress is hindered by the lack of standardized benchmarks. To address this gap, we introduce a comprehensive robotics benchmark, RBench, designed to evaluate robot-oriented video generation across five task domains and four distinct embodiments. By assessing task correctness and visual fidelity through reproducible metrics, our evaluation of 25 models reveals significant deficiencies in generating physically realistic robot behaviors. Furthermore, the benchmark achieves a 0.96 Spearman correlation with human judgment, validating its effectiveness. While RBench provides the necessary lens to identify these deficiencies, achieving physical realism requires moving beyond evaluation to address the critical shortage of high-quality training data. Driven by these insights, we introduce a refined four-stage data pipeline, resulting in RoVid-X, the largest open-source robotic dataset for video generation with 4 million annotated video clips, covering thousands of tasks and enriched with physical property annotations. Extensive experiments demonstrate that finetuning on RoVid-X yields consistent performance gains. Collectively, this synergistic ecosystem of evaluation and data establishes a robust foundation for rigorous assessment and scalable training of video models, accelerating the evolution of embodied AI toward physical intelligence. The code and video demos are available in the supplementary materials.

AI systems can now create impressive videos, but videos that look realistic at first glance may still fail in ways that matter for robots: a robot arm may float, miss an object, skip an important step, or complete a task in a physically impossible way. This makes it hard to know whether today’s video-generation systems can actually help robots learn about the real world. In this work, we build RBench, a new set of tests for robot-related videos. It checks whether generated videos not only look good, but also show robots carrying out the requested tasks in a believable and complete way. We use RBench to evaluate 25 leading video-generation systems and find that many still struggle with realistic robot behavior, even when their videos appear visually polished. To help improve these systems, we also create RoVid-X, a large collection of 4 million robot video clips with detailed descriptions. Training with this data consistently improves robot video generation. Together, RBench and RoVid-X give researchers better tools to measure progress and build video models that better understand physical actions.