Learning Latent Action World Models in the Wild

Agents that can reason and plan in the real world must be able to predict the consequences of their actions. World models possess this capability but require action annotations that can be complex to obtain at scale. Latent action models address this issue by learning an action space from videos alone. Our work studies the training of latent action world models on in-the-wild videos, expanding the scope of existing works that focus on simple robotics simulations, video games, or manipulation data. While diverse videos enable modeling richer actions, they introduce challenges of environmental noise and lack of a common embodiment across videos. To address these, we carefully study the design and evaluation of latent actions. We find that constrained continuous latent actions are better suited for complex in-the-wild videos, compared to vector quantization. For example, actions specific to in-the-wild videos such as humans entering the room, can be modeled and then transferred across videos. However, in the absence of a common embodiment, learned latent actions are localized in space, relative to the camera. Nonetheless, we are able to train a controller that maps known actions to latent ones, allowing us to use latent actions as a universal interface to solve planning tasks on par with action-conditioned baselines.

To reason in the real world, AI agents must predict how their actions change the environment. Usually, this requires training models on videos with precise action labels, which are complex and costly to obtain at scale. Furthermore, most online videos lack these labels entirely. We trained an AI to automatically discover "latent" actions by watching diverse, unlabeled internet videos. By analyzing frame-to-frame changes, the system learns an abstract language of motion. We limited the information these latent actions hold to prevent the AI from simply copying the future frame. This lets the model understand complex events—like someone entering a room—without a predefined action space. This abstract action language can then serve as a universal interface to interact with the AI system. By training a robot to map its physical movements to these latent actions, it can plan and perform tasks on par with models trained on heavily annotated data. This proves we can build general world models from natural, unlabeled videos.