MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction

Latent actions learned from diverse human videos serve as pseudo-labels for vision-language-action (VLA) pretraining, but provide effective supervision only if they remain informative about the underlying ground-truth actions. For effective supervision, latent actions should contain information about the underlying actions even though they are inaccessible. We propose Multi-ViewPoint Latent Action Model (MVP-LAM), which learns latent actions that are highly informative about ground-truth actions from multi-view videos. MVP-LAM trains latent actions with a cross-viewpoint reconstruction objective, so that a latent action from one view must explain the future in another view, reducing reliance on viewpoint-specific cues. On Bridge V2, MVP-LAM produces more action-centric latent actions, achieving higher mutual information with ground-truth actions and improved action prediction, including under out-of-distribution evaluation. Finally, pretraining VLAs with MVP-LAM latent actions improves downstream manipulation performance on various benchmarks. The code and trained checkpoints are available at https://jm-this.github.io/mvp_lam/.

Training a robot to perform everyday manipulation tasks typically requires demonstrations in which a person guides the robot and records each action. Collecting such data at scale is expensive, which limits how broadly robots can learn. Videos of people manipulating objects are far more abundant, but they do not record what action the person took at each moment, so they cannot be used directly for training. Recent methods address this by inferring substitute action labels from how the scene changes between consecutive video frames, and then using these labels to pretrain robot models. A persistent obstacle is that scene changes reflect not only the person's action but also incidental factors such as camera motion and viewpoint shifts. As a result, the inferred labels often encode properties of the camera rather than the underlying action, which weakens their usefulness for training robots. We propose a method that observes the same moment from two synchronized viewpoints and requires the action inferred from one viewpoint to also explain the future observed in the other. Information specific to a single viewpoint cannot satisfy this requirement and is suppressed. Robots pretrained with the resulting labels achieve higher accuracy on standard manipulation benchmarks while requiring substantially less data.