From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models

Latent actions serve as an intermediate representation that enables consistent modeling of vision-language-action (VLA) models across heterogeneous datasets. However, approaches to supervising VLAs with latent actions are fragmented and lack a systematic comparison. This work structures the study of latent action supervision from two perspectives: (i) regularizing the trajectory via image-based latent actions, and (ii) unifying the target space with action-based latent actions. Under a unified VLA baseline, we instantiate and compare four representative integration strategies. Our results reveal a formulation-task correspondence: image-based latent actions benefit long-horizon reasoning, whereas action-based latent actions excel at complex motor coordination. Furthermore, we find that directly supervising the VLM with discrete latent action tokens yields the most effective performance. Finally, our experiments offer initial insights into the benefits of latent action supervision in mixed-data, suggesting a promising direction for VLA training.

Robots are increasingly trained with vision-language-action models, which learn to act from camera observations and human instructions. However, robot training data often comes from different platforms, tasks, and formats, making it difficult for one model to learn a consistent notion of “what action to take next.” This paper studies whether latent actions—compact intermediate descriptions of motion—can help bridge this gap. Instead of proposing one new model, we systematically compare several ways of using latent actions under the same vision-language-action framework. We study latent actions learned from visual changes between images, which can act like high-level plans, and latent actions learned from robot movements, which can turn continuous control commands into discrete tokens. Our experiments show that image-based latent actions are especially helpful for long-horizon tasks and scene generalization, while action-based latent tokens are more effective for motorically complex tasks. We also find that directly training the model to predict discrete latent action tokens is generally more effective than aligning continuous representations. These findings provide practical guidance for building more general and reliable robot learning systems from heterogeneous data.