Dismantling the Illusion of Vision-Language-Action Models Competence via Explicit Distributional Shifts

Given that simulation can never exhaustively enumerate reality, generalization is the determining factor for whether Vision-Language-Action (VLA) models can translate benchmark success into real-world functionality. However, current evaluation protocols often incentivize mechanical memorization rather than robust policy learning, leading to a paradoxical duality of failure: high-scoring models exhibit spurious invariance to semantic changes while simultaneously displaying extreme brittleness to trivial environmental perturbations. To address this, we introduce LIBERO-Gen, a diagnostic benchmark systematically designed to shift evaluation from intuition-driven heuristics to explicit distributional assumptions. Through a hierarchical protocol spanning In-distribution, Compositional, and Domain Generalization, LIBERO-Gen reveals performance stratifications previously masked by standard metrics. Our analysis identifies Pi0.5 as the top performer (64.0% in Spatial-CG; 21.2% in Task-CG). By identifying perceptual instability and action binding collapse as primary failure modes while validating the efficacy of structured ``Stair” sampling, LIBERO-Gen establishes a rigorous baseline for deployment reliability.

Robots are often tested in simulation before being used in the real world, but simulated environments can never cover every situation a robot may face. This paper studies whether AI systems that control robots from camera observations and language instructions can still work when tasks, objects, or environments change in reasonable ways. We introduce LIBERO-Gen, a new evaluation benchmark designed to test this ability more carefully. Instead of only asking whether a model succeeds on familiar benchmark tasks, LIBERO-Gen checks whether the model can handle new object layouts, new task combinations, and changes in the surrounding environment. Our results show that high scores on standard benchmarks do not always mean that a model has learned robust robot skills. Some models appear to rely on shortcuts from the training data: they may ignore meaningful changes in the task, while also failing when small details in the scene are changed. Among the tested models, Pi0.5 performs best overall, but it still shows clear weaknesses under harder generalization settings. We also identify common failure patterns, such as unstable visual understanding and incorrect matching between instructions and actions. Overall, LIBERO-Gen provides a more careful way to evaluate robot generalization in constrained benchmark settings, but it is intended to complement, not replace, real-world robot testing.