3DPoV: Improving 3D understanding via Patch Ordering on Videos

Visual foundation models have achieved remarkable progress in scale and versatility, yet understanding the 3D world remains a fundamental challenge. While 2D images contain cues about 3D structure that humans readily interpret, deep models often fail to exploit them, underperforming on tasks such as multiview semantic consistency--crucial for applications including robotics and autonomous driving. We propose a self-supervised approach to enhance the 3D understanding of vision foundation models by (i) introducing a temporal nearest-neighbor consistency loss that finds corresponding points across video frames and enforces consistency between their nearest neighbors, (ii) incorporating reference-guided ordering that requires patch-level features to be not only expressive but also consistently aligned, and (iii) constructing a mixture of video datasets tailored to these objectives, thereby leveraging rich 3D information. Our method, 3DPoV, achieves state-of-the-art performance in keypoint matching under viewpoint variation, as well as in depth and surface normal estimation, and consistently improves a diverse set of backbones, including DINOv3.

Humans can look at an image and intuitively understand its 3D structure: recognizing which elements are closer to the camera, or identifying the same object from a different angle. Modern AI vision systems still struggle with this. While they excel at associating visual elements even across very different contexts, this ability degrades significantly under viewpoint changes, because these models learn patterns tied to a specific 2D perspective without any grounding in 3D geometry. We introduce 3DPoV, a method that teaches vision models to better understand the 3D world by training on videos. The key idea is simple: if two frames show the same scene from different angles, patches corresponding to the same physical location should be similar to the same set of reference patches. We enforce this by tracking points through video and requiring the model to preserve the ordering of feature similarities over time. The result is a lightweight fine-tuning approach, requiring just 20 GPU hours and using a mix of object-centric and scene-based videos, that consistently improves existing vision models on keypoint matching, depth estimation, and surface normal prediction, with uniform gains across viewpoint variation, occlusion, and lighting changes.