CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation

Code-as-Policy (CaP) is a paradigm in which a language or vision-language model generates executable robot control programs, yet its effectiveness as an autonomous controller for embodied manipulation remains underexplored. Prior CaP systems often rely on high-level, human-designed primitives, making it difficult to separate agent capability from designer-provided scaffolding. We present CaP-X, an open-access framework for systematically studying Code-as-Policy agents in robot manipulation. CaP-X includes four components. CaP-Gym is an interactive environment in which coding agents control robots by synthesizing and executing programs that compose perception and control primitives. Building on this foundation, CaP-Bench evaluates frontier language and vision-language models across varying levels of abstraction, interaction, and perceptual grounding. Across 12 models, the task success rates improve with human-crafted abstractions but degrade as these priors are removed, exposing a dependence on designer scaffolding. At the same time, we observe that scaling test-time computation with multi-turn interaction, structured execution feedback, visual differencing, automatic skill synthesis, and ensembled reasoning can substantially improve robustness even when agents operate over low-level primitives. These findings motivate CaP-Agent0, a training-free framework that achieves near human-level reliability on several manipulation tasks in simulation and on real embodiments. CaP-RL explores reinforcement learning with verifiable rewards to improve success rates and supports sim-to-real transfer through a shared code-as-action-space interface. Together, CaP-X provides an open-access platform for advancing embodied coding agents. Project page: https://capgym.github.io

Robots are becoming more capable, but it is still difficult to make them reliably perform everyday manipulation tasks, such as picking up, placing, stacking, or using objects. One promising idea is to let AI systems write programs that control robots, rather than asking them to directly output every robot movement. However, many prior systems rely on human-written code, making it unclear whether the AI system is truly capable or mostly benefiting from human-engineered tools. We introduce CaP-X, a framework for studying and improving AI systems that write code to control robots. CaP-X lets researchers test these systems across many robot tasks, from easier settings with helpful built-in tools to harder settings where the AI must compose lower-level robot skills on its own. We find that today’s strongest AI models still depend heavily on human-provided structure, but they become much more reliable when they can interact with the environment, inspect failures, reason over visual changes, reuse useful skills, and learn from trial-and-error. CaP-X provides an open platform for measuring progress toward AI agents that can write reliable, inspectable robot-control programs and transfer them from simulation to real robots.