Seeing Realism from Simulation: Efficient Video Transfer for Vision-Language-Action Data Augmentation

Vision-language-action (VLA) models typically rely on large-scale real-world videos, whereas simulated data, despite being inexpensive and highly parallelizable to collect, often suffers from a substantial visual domain gap and limited environmental diversity, resulting in weak real-world generalization. We present an efficient video augmentation framework that converts simulated VLA videos into realistic training videos while preserving task semantics and action trajectories. Our pipeline extracts structured conditions from simulation via video semantic segmentation and video captioning, rewrites captions to diversify environments, and uses a conditional video transfer model to synthesize realistic videos. To make augmentation practical at scale, we introduce a diffusion feature-reuse mechanism that reuses video tokens across adjacent timesteps to accelerate generation, and a coreset sampling strategy that identifies a compact, non-redundant subset for augmentation under limited computation.Extensive experiments on Robotwin 2.0, LIBERO, LIBERO-Plus, and a real robotic platform demonstrate consistent improvements.For example, our method improves RDT-1B by 8\% on Robotwin 2.0, and boosts $\pi_0$ by 5.1\% on the more challenging LIBERO-Plus benchmark. Code is available at: https://github.com/nanfangxiansheng/Seeing-Realism-from-Simulation

Training robots to act reliably in the real world usually requires many real-world videos, which are expensive and slow to collect. Computer simulations can generate robot videos much more cheaply, but simulated videos often look too clean, repetitive, or unrealistic, so robots trained on them may not work well in real environments. This paper introduces a way to turn simulated robot videos into realistic training videos while keeping the original task and robot motion unchanged. Our method first identifies what is happening in the simulated video, then describes the scene in natural language, changes the description to create more diverse environments, and finally generates a realistic version of the video that still follows the same action. To make this practical for large datasets, we also reduce unnecessary video generation and speed up the generation process. Experiments across several robot-learning benchmarks and a real robot show that the resulting videos help robots perform tasks more successfully and transfer better from simulation to the real world.