Rethinking Temporal Consistency in Video Object-Centric Learning: From Prediction to Correspondence

The de facto approach in video object-centric learning maintains temporal consistency through learned dynamics modules that predict future object representations, called slots. We demonstrate that these predictors function as expensive approximations of discrete correspondence problems. Modern self-supervised vision backbones already encode instance-discriminative features that distinguish objects reliably. Exploiting these features eliminates the need for learned temporal prediction. We introduce Grounded Correspondence, a framework that replaces learned transition functions with deterministic bipartite matching. Slots initialize from salient regions in frozen backbone features. Frame-to-frame identity is maintained through Hungarian matching on slot representations. The approach requires zero learnable parameters for temporal modeling yet achieves competitive performance on MOVi-D, MOVi-E, and YouTube-VIS. Project page: https://magenta-sherbet-85b101.netlify.app/

Videos are hard for AI systems because they must not only find objects in each frame, but also keep the same identity for each object as it moves. Most object-centric video models solve this by training an extra module to predict the next state of each object representation, or “slot.” We show that this prediction step is often unnecessary when the video model uses a strong pretrained vision backbone: the backbone’s features already contain enough information to tell objects apart. Our method, Grounded Correspondence, starts each slot from image regions that look like object centers in these features, then matches slots between neighboring frames with a standard, deterministic assignment algorithm. In other words, we replace learned future prediction with direct object correspondence. This removes all learnable parameters from the temporal part of the model while still maintaining object identities across time. In experiments on MOVi-D, MOVi-E, and YouTube-VIS, the method improves object grouping on synthetic videos and remains competitive on real-world videos. The result suggests that simpler and cheaper video-understanding models are possible, although the method still has limitations, such as long occlusions and fixed object counts.