From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation

Video generation models offer a promising imagination mechanism for robot manipulation by predicting long-horizon future observations, but effectively exploiting these imagined futures for action execution remains challenging. Existing approaches either condition policies on predicted frames or directly decode generated videos into actions, both suffering from a mismatch between visual realism and control relevance. As a result, predicted observations emphasize perceptual fidelity rather than action-centric causes of state transitions, leading to indirect and unstable control. To address this gap, we propose MoLA (Mixture of Latent Actions), a control-oriented interface that transforms imagined future videos into executable representations. Instead of passing predicted frames directly to the policy, MoLA leverages multiple pretrained inverse dynamics models to infer a mixture of latent actions implied by generated visual transitions. These modality-aware inverse dynamics models capture complementary semantic, depth, and flow cues, providing a structured and physically grounded action representation that bridges video imagination and policy execution. We evaluate our approach on simulated benchmarks (LIBERO, CALVIN, and LIBERO-Plus) and real-world robot manipulation tasks, achieving consistent gains in task success, temporal consistency, and generalization.

Robots are increasingly expected to follow human instructions and manipulate objects in homes, labs, and factories. One promising way to improve robot decision making is to let a model imagine what might happen next, for example by predicting future video frames. However, these imagined images are not directly the same as actions that a robot can execute. In this work, we propose MoLA, a method that turns imagined future videos into action-centered guidance for robot control. Instead of asking the robot policy to interpret generated images alone, MoLA uses several models to estimate what movements are implied by changes in the scene, using complementary cues such as object meaning, depth, and motion. These compact action hints are then used to guide the robot’s control policy. We test the method in standard simulation benchmarks and real-world robot manipulation tasks. MoLA improves task success and generalization, showing that imagined futures are more useful when they are translated into representations that are closer to robot actions.