LARA: Latent Action Representation Alignment for Vision-Language-Action Models

Visual-language-action (VLA) models enable robots to predict actions directly from observations and language instructions, but their performance depends on large-scale, high-quality data and is limited by the scarcity of real-world robot action datasets. To facilitate VLA model learning with abundant unlabeled human videos, Latent Action Models (LAM) learn latent action representations from visual dynamics to provide additional supervision for VLA learning. However, LAM and VLA are typically trained separately, leaving LAM ungrounded during VLA training and VLA models constrained by frozen LAM representations. To address these issues, we propose Latent Action Representation Alignment (LARA), a plug-and-play framework that jointly optimizes LAM and VLA via representation alignment. This enables reciprocal benefits where LAMs learn with action trajectories to avoid spurious visual changes, while VLAs are regularized by forward dynamics learned within LAMs to reduce hallucinations of functionally ineffective trajectories. We demonstrate LARA's versatility and effectiveness for pre-training, post-training enhancement of pre-trained VLA models, and LAM refinement, achieving an average of ~10%, ~5%, and ~15% improvement over 3 simulation and 1 meticulously designed real-world robotic manipulation benchmarks. The code is publicly available at https://github.com/lmy1001/LARA.

Visual-language-action (VLA) models enable robots to predict actions directly from observations and language instructions, but their performance depends on large-scale, high-quality data and is limited by the scarcity of real-world robot action datasets. In contrast, videos of people and robots performing everyday tasks are widely available, but they usually do not contain robot action labels. This paper introduces LARA, a method that helps robots learn from both labeled robot data and unlabeled videos. LARA connects a model that learns hidden action information from videos with a robot policy model that predicts real robot actions. Instead of training these two models separately, LARA trains them together so that they can improve each other: the video-based model learns action-relevant motion, while the robot policy becomes better grounded in how actions change the world. Experiments across simulation and real-world robot manipulation tasks show that LARA improves robot performance when used for full training, post-training enhancement of existing models, and refinement of latent action representations. These results suggest that LARA is a practical and flexible way to make better use of large-scale video data for general-purpose robot learning.