GeoMoLa: Geometry-Aware Motion Latents for Learning Robust Manipulation Policies

Learning motion latents for robotic manipulation heavily relies on extracting motion patterns from visual sequences, yet effective action abstractions require understanding three-dimensional geometric transformations. Here, we introduce GeoMoLa (Geometry-Aware Motion Latents), which learns discrete motion latent codes by predicting how point clouds evolve during manipulation rather than reconstructing visual observations. This four-dimensional objective – spatial geometry changing through time – forces latent representations to encode actual physical motion rather than appearance patterns. GeoMoLa achieves state-of-the-art performance using only single-view RGB-D input, while existing methods require multi-view reconstruction, succeeding across diverse manipulation benchmarks. Our ablations reveal that geometric prediction is the key to driving performance, quantitatively validating that manipulation depends on spatial understanding. Furthermore, the learned codes exhibit effective motion abstraction: applying them to novel scenes produces physically consistent transformations regardless of visual context. Our real-world experiments also confirm this robustness capability, achieving robust manipulation with minimal demonstrations in cluttered environments where geometric reasoning determines success. Thus, we demonstrate that effective motion latents for robot control can better emerge from understanding motion through its three-dimensional effects rather than pixel-level patterns.

Robots learn many skills by identifying reusable patterns in how actions change the world. Most existing methods learn these patterns from ordinary videos, focusing on how scenes look rather than how objects are arranged and move in three-dimensional space. This can make robots struggle when objects appear from new viewpoints, become partially hidden, or are placed in unfamiliar environments. We developed a new approach that teaches robots to understand how the 3D structure of a scene changes over time during manipulation. Instead of learning from visual appearance alone, our method learns to predict how objects move and transform in space as actions are performed. This encourages the robot to capture the underlying geometric relationships that determine whether a task succeeds. Experiments across simulation and real-world settings show that robots trained with these geometry-aware motion representations perform tasks more reliably, especially in cluttered scenes where precise spatial reasoning is essential. Our results suggest that helping robots understand how the world changes in 3D may be more important than improving visual realism alone, opening a new direction for robot learning.