Co-Evolving Latent Action World Models

Adapting pre-trained video generation models into controllable world models via latent actions is a promising step towards creating generalist world models. The dominant paradigm adopts a two-stage approach that trains latent action model (LAM) and the world model separately, resulting in redundant training and limiting their potential for co-adaptation. A conceptually simple and appealing idea is to directly replace the forward dynamic model in LAM with a powerful world model and training them jointly, but it is non-trivial and prone to representational collapse. In this work, we propose CoLA-World, which for the first time successfully realizes this synergistic paradigm, resolving the core challenge in joint learning through a critical warm-up phase that effectively aligns the representations of the from-scratch LAM with the pre-trained world model. This unlocks a co-evolution cycle: the world model acts as a knowledgeable tutor, providing gradients to shape a high-quality LAM, while the LAM offers a more precise and adaptable control interface to the world model. Empirically, CoLA-World matches or outperforms prior two-stage methods in both video simulation quality and downstream visual planning, establishing a robust and efficient new paradigm for the field.

Creating intelligent agents that can operate anywhere requires world models as internal simulators that help AI understand how an environment works. A major challenge is learning how to control these simulators using abstract actions learned purely from watching videos, rather than relying on predefined joysticks or buttons. Traditionally, researchers trained an action-learning model and the world simulator separately in two distinct stages, which is inefficient and limits their ability to adapt to each other. To solve this, we introduce CoLA-World, a new method that trains both the action model and the world simulator simultaneously. To prevent the training process from crashing, a common issue when combining them directly, we designed a brief "warm-up" phase to help the action model synchronize with the simulator before fully merging their learning processes. This joint approach unlocks a "co-evolution" cycle: the powerful simulator acts as a tutor to help define better actions, while the improved actions give the simulator more precise controls. Ultimately, this creates higher-quality video predictions and helps AI agents plan physical tasks much more effectively.