Multi-view Consistent Latent Action Learning for World Modeling and Control

The scalability of world models is currently bottlenecked by the scarcity of action annotations. While self-supervised latent action learning offers a potential solution, existing single-view paradigms—relying on information bottlenecks or Vector Quantization (VQ)—often conflate superficial 2D pixel displacements with the underlying physical-spatial dynamics of an action. Consequently, these methods remain highly susceptible to view-dependent noise, such as camera shake. We introduce MuCoLA (Multi-view Consistent Latent Action learning), a framework that learns robust, view-invariant action representations by enforcing semantic consistency across synchronized video streams. MuCoLA utilizes a Student-Teacher network with DINO-style self-distillation to align action distributions across viewpoints, effectively filtering high-frequency visual noise while preserving motion semantics. Theoretical analysis reveals that our multi-view objective functions as a spectral filter, isolating agent dynamics from environmental nuisances. Empirically, MuCoLA significantly outperforms baselines in action regression, video reconstruction, and downstream visual control tasks. Furthermore, we demonstrate that MuCoLA exhibits favorable scaling properties with respect to model capacity and data volume, paving the way for large-scale action-free world modeling.

We propose a new method, MuCoLA, that enables AI systems to learn the nature of actions from videos more accurately. At present, many AI models only learn from videos from a single perspective, and it is easy to mistake unrelated visual noise such as camera shake and light change as the motion of the object or the robot itself, this leads to inaccurate and unreliable representation of learned actions. Mucola's core idea is to mimic humans: when we view the same action from different angles, we can ignore perspective differences and grasp its physical nature. To this end, MuCoLA simultaneously analyzes multiple video streams (e.g., front and side views of a task) and requires the AI to extract consistent action information across different views. Through a network of teachers and students learning from each other, the system can automatically filter out visual distractions caused by differences in perspective and focus on learning the real, perspective-independent motion patterns of objects or robots. Theoretical analysis shows that the multi-view consistency target acts like a filter, effectively separating the real action signal. Experiments show that based on the motion representation learned by MuCoLA, AI can predict the future picture, perform visual planning and control the robot to complete complex tasks such as grasping and pushing objects, the performance is significantly better than previous methods and generalizes to angles never seen before. This work is an important step towards building more robust and controllable world models and agents.