Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action

We introduce SOMA, the Spatial Memory framework for Out-of-Vision Manipulation in Vision-Language-Action (VLA) models. Most existing VLAs implicitly assume that task-relevant objects are always visible, leading to brittle and reactive behaviors when targets fall outside the camera’s field of view. SOMA addresses this limitation by equipping VLAs with a persistent spatial memory constructed from multi-view observations acquired via a movable head camera, enabling reasoning beyond the current visual frustum. The framework consists of three components: Spatial Memory Construction, which aggregates angular-wise observations into a unified spatial–semantic representation through scanning; Dynamic Memory Refinement, which maintains global consistency over time; and Contextual Memory Retrieval, which activates instruction-relevant spatial cues during manipulation. We evaluate SOMA on five challenging real-world out-of-vision manipulation tasks, including multi-step and dual-arm scenarios where target objects are initially invisible. Experimental results show that SOMA not only improves task success rates, but also induces qualitatively different manipulation behaviors, with faster target localization, reduced viewpoint search, and near one-shot grasping under partial observability. Additional experiments on RoboCasa GR1 and SimplerEnv further validate the effectiveness of SOMA’s memory design under conventional fully observable settings. Code will be released soon.

When you ask a robot to "pick up the cup and put it in the basket," it uses a camera to find the cup. But today's instruction-following robots can only act on what their camera sees at that instant. If the cup sits just out of view, the robot has no sense that it still exists nearby, so it freezes before it can even begin. We gave robots a form of spatial memory. Before acting, the robot turns its movable head camera to scan the workspace and build a map of where each object is — much as a person remembers where they set down their keys. As it works, it keeps this map up to date and recalls the relevant part whenever the instruction calls for it, letting it reach for things it cannot currently see. In tests on a real two-armed robot and in simulations, this memory helped robots locate targets faster, search around far less, and usually grasp on the first try. That kind of reliability matters for robots in homes and factories, which can rarely see everything at once and must act on what they remember.