DiLA: Disentangled Latent Action World Models

Latent Action Models (LAMs) enable the learning of world models from unlabeled video by inferring abstract actions between consecutive frames. However, LAMs face a fundamental trade-off between action abstraction and generation fidelity. Existing methods typically circumvent this issue by using two-stage training with pre-trained world models or by limiting predictions to optical flow. In this paper, we introduce DiLA, a novel Disentangled Latent Action world model that aims to resolve this trade-off via content-structure disentanglement. Our key insight is that disentanglement and latent action learning are co-evolving: the predictive bottleneck inherent in latent action learning serves as a driving force for disentanglement, compelling the model to distill spatial layouts into the structure pathway while offloading visual details to a separate content pathway for generation. This synergy yields a continuous, semantically structured latent action space without compromising generative quality. DiLA achieves superior results in video generation quality, action transfer, visual planning, and manifold interpretability. These findings establish DiLA as a unified framework that simultaneously achieves high-level action abstraction and high-fidelity generation, advancing the frontier of self-supervised world model learning.

Modern AI systems often learn by watching videos, but videos usually do not come with labels telling the system what action caused each change. Our work asks whether an AI can discover these hidden actions on its own. We propose DiLA, a model that learns from ordinary videos by separating “what is moving” from “what things look like.” For example, when a hand moves a cup, the action should describe the movement, not the cup’s color, the table texture, or the lighting. DiLA learns this separation automatically: one part of the model focuses on motion and spatial changes, while another part keeps track of visual appearance. This helps DiLA learn actions that are more general and reusable. An action learned from one video, such as a person picking something up, can be transferred to a different setting, such as a robot arm performing a similar movement. The learned actions also form meaningful patterns that reflect real physical changes like moving, turning, or scaling. Finally, we show that these learned video-based actions can help robots plan future behavior more effectively.