Any3D-VLA: Enhancing VLA Robustness via Diverse Point Clouds

Existing Vision-Language-Action (VLA) models typically take 2D images as visual input, which limits their spatial understanding in complex scenes. How can we incorporate 3D information to enhance VLA capabilities? We conduct a pilot study across different observation spaces and visual representations. The results show that explicitly lifting visual input into point clouds yields representations that better complement their corresponding 2D representations. To address the challenges of (1) scarce 3D data and (2) the domain gap induced by cross-environment differences and depth-scale biases, we propose Any3D-VLA. It unifies the simulator, sensor, and model-estimated point clouds within a training pipeline, constructs diverse inputs, and learns domain-agnostic 3D representations that are fused with the corresponding 2D representations. Simulation and real-world experiments demonstrate Any3D-VLA's advantages in improving performance and mitigating the domain gap. Our project homepage is available at https://xianzhefan.github.io/Any3D-VLA.github.io.

(1) Problem: Vision-Language-Action (VLA) models, trained on massive collections of action trajectories paired with language instructions, hold great promise for achieving general-purpose embodied intelligence. While these models are increasingly powerful in language and semantic understanding, their spatial understanding is still largely inherited from 2D visual backbones. As a result, they are particularly brittle in scenarios involving small objects, viewpoint changes, and occlusions. (2) Solution: We propose Any3D-VLA, a plug-in pipeline for existing VLA backbones. Given RGB images with optional depth, we first lift the visual input to point clouds and compress them. A pre-trained point cloud encoder then produces point-wise embeddings. Finally, we align and fuse the 3D embeddings with corresponding 2D patch features, and feed the fused representation into the downstream backbone. We synthesize a large-scale RGBD pre-training dataset for VLA tasks, covering diverse depth sources to construct varied point-cloud inputs. (3) Impact: In real-world deployment, Any3D-VLA does not rely on expensive depth hardware or stringent data-collection conditions; with hybrid point cloud training, it learns more robust 3D geometric reasoning.