VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models

While Vision-Language-Action models (VLAs) are rapidly advancing toward generalist robot policies, quantitatively characterizing their capability boundaries and failure modes remains challenging. To address this, we introduce VLA-Arena, a comprehensive benchmark. It features a novel structured task design framework to quantify difficulty across three orthogonal axes: (1) Task Structure, (2) Language Command, and (3) Visual Observation. This allows us to systematically design tasks with fine-grained difficulty levels, enabling a precise measurement of model capability frontiers. For task structure, VLA-Arena comprises 11 task suites organized into four dimensions: Safety, Distractor, Extrapolation, and Long Horizon, totaling 170 tasks. Each suite spans three difficulty levels (L0–L2), with fine-tuning restricted to L0 to rigorously assess generalization. Orthogonal to this, language (W0-W4) and visual (V0-V4) perturbations can be applied to any task as diagnostic probes to distinguish robust grounding from superficial pattern matching. Our extensive evaluation of state-of-the-art VLAs reveals critical limitations: memorization over generalization, superficial visual perception, and a neglect of safety constraints. Additionally, model rank reversals across L0–L2 validate that each level provides non-redundant insights. To foster research addressing these model limitations and ensure reproducibility, we provide the complete VLA-Arena framework, including an end-to-end toolchain from task definition to automated evaluation and the VLA-Arena-S/M/L datasets for fine-tuning. Our benchmark, datasets, models, and leaderboard are publicly available at https://vla-arena.github.io.

Artificial Intelligence is increasingly being used to build brains for robots, allowing them to see their surroundings, understand human language, and take physical actions. However, before we can safely bring these AI-driven robots into our homes and workplaces, we need to know exactly what they can and cannot do, and how they might fail. Currently, it is very difficult to measure their true limits. To solve this, we created VLA-Arena, a comprehensive and open-source testing ground for robot AI. Instead of just checking if a robot can complete a basic chore, VLA-Arena systematically dials up the difficulty. We test how the AI handles complex environments, unexpected visual changes (e.g., confusing backgrounds or different lighting), rephrased human instructions, and strict safety rules (e.g., avoiding fragile objects). By putting the most advanced robot AI models through VLA-Arena, we discovered several critical blind spots: current models tend to merely memorize their training rather than truly understanding the world, they are easily confused by visual distractions, and most alarmingly, they frequently ignore safety rules just to get the job done. By sharing our complete testing framework with the public, we aim to provide researchers with the tools needed to build smarter, safer, and more reliable robots for the real world.