LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries

Vision-Language-Action (VLA) models have shown promise in robot manipulation but often struggle to generalize to new instructions or complex multi-task scenarios. We identify a critical pathology in current training paradigms where goal-driven data collection creates a dataset bias. In such datasets, language instructions are highly predictable from visual observations alone, causing the conditional mutual information between instructions and actions to vanish, a phenomenon we term Information Collapse. Consequently, models degenerate into vision-only policies that ignore language constraints. To address this, we propose LangForce, enforces instruction following via Bayesian decomposition. By introducing learnable Latent Action Queries, we construct a dual-branch architecture to estimate both a vision-only prior $p(a \mid v)$ and a language-conditioned posterior $\pi(a \mid v, \ell)$. We then optimize the policy to maximize the conditional Pointwise Mutual Information (PMI) between actions and instructions. This objective effectively penalizes the vision shortcut and rewards actions that explicitly explain the language command. Extensive experiments across on three benchmarks demonstrate substantial gains, including an 11.3\% improvement on the challenging OOD SimplerEnv benchmark, validating the ability of LangForce to robustly ground language in action.

Robots are increasingly trained to follow human instructions, such as picking up a specific object or placing it in a certain location. However, many robot training datasets unintentionally make the task too easy to guess from the image alone. For example, if a robot usually sees the same scene whenever it is asked to perform a certain task, it may learn to act based only on visual patterns rather than actually understanding the instruction. This can make the robot appear successful during training, but fail when the same scene could require different actions. We study this problem and propose LangForce, a training method that encourages robots to keep using language when choosing actions. LangForce compares what the robot would infer from the image alone with what it should do when the instruction is also considered. This helps the robot focus on the words that matter for the task. Across simulated and real robot experiments, LangForce improves instruction following, especially when visual cues are ambiguous or misleading.