Memory as Dynamics: Learning Reliability-Guided Predictive Models for Online Video Perception

Predictive memory has recently emerged as a powerful mechanism for online video models, enabling temporal reasoning beyond static memory banks. However, we observe a paradoxical phenomenon in which predictive memory often exacerbates drift under occlusion or rapid motion, as inaccurate predictions contaminate the internal state and lead to irreversible identity loss. We identify this failure as a reliability mismatch: predictive dynamics are applied uniformly despite high uncertainty and weak observations. To address this issue, we reinterpret video memory as a dynamic latent process rather than a static buffer. Building on this insight, we introduce Reliability-Guided Predictive Memory (RPM), a framework that explicitly regulates when and how predictive dynamics should influence online video perception. RPM integrates a latent world model based on state-space dynamics to generate predictive priors, while employing a reliability-aware fusion policy that suppresses unreliable predictions during occlusion. We instantiate RPM on a SAM2-based model and evaluate it on visual object tracking benchmarks. Experiments demonstrate that our method reduces drift after occlusion, consistently outperforming strong baselines that rely on either static memory or unconditional predictive modeling. We further show that this reliability-guided mechanism transfers to video object segmentation across multiple standard benchmarks. These findings establish that predictive memory is beneficial only when its reliability is explicitly modeled, and define a general principle for robust online video perception. The code is available at https://github.com/minwookim01/memory-as-dynamics.

Imagine an AI system watching a video and trying to keep track of one object. When that object is briefly hidden behind another object, the system has to predict where it is likely to be. However, prediction is needed precisely when visual evidence is weakest, so incorrect guesses can be stored as if they were true. Over time, these errors accumulate, causing the tracker to drift away from the real object and eventually lose it. Our method, called RPM, prevents this by checking whether each prediction is reliable before allowing it to influence memory. A prediction is used only when two conditions are satisfied: the current visual evidence is not sufficient on its own, and the prediction is still consistent with the visible parts of the scene. If either condition is not met, the prediction is rejected before it can introduce errors. As a result, RPM helps video trackers follow objects more reliably over long sequences, recover them more effectively after temporary occlusion, and maintain nearly the same running speed. This kind of stability is important for real-time systems such as autonomous vehicles and robots, where a brief loss of visibility should not cause the system to lose track of what is happening.