Factored Latent Action World Models

Learning latent actions from action-free video has emerged as a powerful paradigm for scaling up controllable world model learning. Latent actions provide a natural interface for users to iteratively generate and manipulate videos. However, most existing approaches rely on monolithic inverse and forward dynamics models that learn a single latent action to control the entire scene, and therefore struggle in complex environments where multiple entities act simultaneously. This paper introduces Factored Latent Action Model (FLAM), a factored dynamics framework that decomposes the scene into independent factors, each inferring its own latent action and predicting its own next-step factor value. This factorized structure enables more accurate modeling of complex multi-entity dynamics and improves video generation quality in action-free video settings compared to monolithic models. Based on experiments on both simulation and real-world multi-entity datasets, we find that FLAM outperforms prior work in prediction accuracy and representation quality, and facilitates downstream policy learning, demonstrating the benefits of factorized latent action models.

Many useful videos, such as traffic scenes, games, and robot demonstrations, show several things moving at once, but they usually do not come with labels saying which action each agent takes. Existing video-learning systems often try to summarize the entire scene change with one hidden "action", which is like describing all cars in an intersection with one command. We introduce Factored Latent Action Models, or FLAM, which instead breaks a scene into separate moving parts and learns a hidden action for each part directly from video. This lets the model predict what happens next while still accounting for interactions among objects, such as cars responding to nearby cars. Across simulated worlds and real driving videos, FLAM produced more accurate future videos than earlier approaches that use a single scene-level action. The learned actions can also be used to edit one part of a generated video while leaving other parts mostly unchanged, and they can help train decision-making systems when only a small amount of action-labeled data is available. This work is a step toward AI systems that can learn controllable world models from ordinary videos, without requiring every action in those videos to be manually labeled.