G$^2$TAM: Geometry Grounded Track Anything Model

Human spatial understanding arises from jointly perceiving geometry and semantics, enabling consistent object identification and localization across viewpoints and time. Current video segmentation models depend on explicit object appearance memory banks for instance tracking, yet they remain vulnerable to large viewpoint changes and long-term occlusions. Leveraging the spatial consistency afforded by modern feed-forward 3D reconstruction models, we propose the Geometry Grounded Tracking Anything Model (G$^2$TAM), a unified framework for promptable instance tracking in 3D using only unordered RGB images or videos. G$^2$TAM employs spatially aligned geometric representations as implicit memory, ensuring stable instance identity and localization across frames and views. At its core is a cross-modal spatial encoder that integrates visual and textual prompts into a shared geometric space, enabling end-to-end spatial reconstruction and instance-consistent mask prediction. To support training and evaluation, we construct InsTrack, a large-scale dataset with a dedicated validation split for benchmarking. Extensive experiments show that G$^2$TAM delivers strong cross-view consistency, promptable instance spatial tracking, video object segmentation, and spatial reconstruction, establishing a foundation for interactive, geometry-grounded spatial reasoning.

AI systems are increasingly able to segment and track objects in videos, but they often rely on how objects look from frame to frame. This can fail when the camera moves a lot, when an object is hidden for a long time, or when the same object looks very different from another viewpoint. In this work, we ask whether 3D spatial understanding can help an AI system remember and track objects more reliably. We introduce G2TAM, a model that jointly reconstructs scene geometry and segments a prompted object across images or videos. Instead of storing only visual appearance as memory, the model uses a geometry-aware representation as a more stable form of spatial memory. A user can specify the target object with a point, box, or text description, and the model predicts consistent object masks across different views and time steps. This matters because reliable object tracking is important for applications such as robotics, augmented reality, and interactive scene understanding. By connecting segmentation with 3D geometry, our work moves toward AI systems that understand not only what objects look like, but also where they are in the surrounding space.