Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models

Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is constrained by the high cost and safety risks of real-world interaction. Training VLA models in interactive world models avoids these issues but introduces several challenges, including pixel-level world modeling, multi-view consistency, and compounding errors under sparse rewards. Building on recent advances across large multimodal models and model-based RL, we propose VLA-MBPO, a practical framework to tackle these problems in VLA finetuning. Our approach has three key design choices: (i) adapting unified multimodal models (UMMs) for data-efficient world modeling; (ii) an interleaved view decoding mechanism to enforce multi-view consistency; and (iii) chunk-level branched rollout to mitigate error compounding. Theoretical analysis and experiments across simulation and real-world tasks demonstrate that VLA-MBPO significantly improves policy performance and sample efficiency. Crucially, our method maintains a universal set of hyperparameters across all tasks, underscoring its robustness and scalability for real-world robotic deployment.

Many modern robots learn skills through reinforcement learning, where they improve by trial and error. However, this process often requires an enormous number of real-world interactions, making training expensive, time-consuming, and sometimes unsafe. A promising alternative is to train robots inside AI-generated “imagined” environments, known as world models. Yet these simulated environments can produce small prediction mistakes that accumulate over time, causing robots to learn incorrect behaviors. We introduce VLA-MBPO, a new framework that enables Vision-Language-Action (VLA) robot models to learn more reliably inside AI-generated simulations. Our approach uses a unified multimodal world model that jointly predicts future robot observations and task rewards from images, language instructions, and actions. We also develop techniques that maintain consistency across multiple camera views and reduce the accumulation of simulation errors during long-horizon tasks. We evaluate our method on both simulated and real-world robotic tasks, including folding towels, inserting cables\, and whole-body mobile manipulation. VLA-MBPO improves both learning efficiency and task success rates while requiring far fewer real-world interactions. These results bring scalable and practical robot learning closer to real-world deployment.