GAE: Unleashing Physical Potential of VLM with Generalizable Action Expert

Vision-language models demonstrate strong reasoning and planning abilities, yet grounding these predictions into precise robot actions remains a central challenge. Existing Vision-Language-Action methods typically entangle reasoning and action generation, leading to limited generalization and costly adaptation. We propose to learn a \textbf{G}eneralizable \textbf{A}ction \textbf{E}xpert (\textbf{GAE}), a task-agnostic model that converts sparse geometric plans into dense robot actions. Our approach introduces a sparse geometric interface: the VLM predicts sparse 3D waypoints representing high-level intention, while GAE maps these waypoints together with real-time point cloud observations to continuous action trajectories. GAE is pretrained on a large-scale pointcloud–trajectory dataset comprising \textbf{150k} trajectories from both simulation and real-world robots. To further improve efficiency and generalization, we introduce an \textbf{Action Pre-training, Pointcloud Fine-tuning (APPF)} scheme that decouples learning action dynamics from geometry grounding. After pretraining, GAE is frozen and reused across downstream tasks, requiring only lightweight fine-tuning of the VLM to produce the sparse interface. Extensive experiments show that our method achieves strong performance and generalization across diverse visual domains, camera viewpoints, and natural language instructions.

Robots are becoming better at understanding images and language, but it is still difficult to turn a high-level instruction into precise physical movements. For example, a robot may understand what object to move and where it should go, but it still needs to decide the detailed motions needed to complete the task safely and accurately. We propose GAE, a general action expert that helps bridge this gap. Instead of asking one model to both reason about the task and control the robot, our method separates these two steps. A vision-language model first predicts a small number of 3D points that describe the intended motion, and GAE then converts these points, together with the robot’s current visual observations, into detailed robot actions. GAE is trained on a large collection of robot motion data from both simulation and real-world robots. Once trained, it can be reused for many different tasks, so adapting the system to a new task only requires lightweight training of the vision-language model. Experiments show that this approach works well across different scenes, camera views, visual styles, and natural language instructions, making robot control more flexible and easier to generalize.