CAST: Modeling Visual State Transitions for Consistent Video Retrieval

As video content creation shifts toward long-form narratives, composing short clips into coherent storylines becomes increasingly important. However, prevailing retrieval formulations remain context-agnostic at inference time, prioritizing local semantic alignment while neglecting the state and identity consistency. To address this structural limitation, we formalize the task of Consistent Video Retrieval (CVR) and introduce a diagnostic benchmark spanning YouCook2, COIN, and CrossTask. We propose CAST (Context-Aware State Transition), a lightweight, plug-and-play adapter compatible with diverse frozen vision-language embedding spaces. By predicting a state-conditioned residual update (Δ) from visual history, CAST introduces an explicit inductive bias for latent state evolution. Extensive experiments show that CAST improves performance on YouCook2 and CrossTask, remains competitive on COIN, and consistently outperforms zero-shot baselines across diverse foundation backbones. Furthermore, CAST provides a useful reranking signal for black-box video generation candidates (e.g., from Veo), promoting more temporally coherent continuations.

Many video tools need to choose or generate the next clip in a longer story, such as the next step in a cooking tutorial or a repair video. Current systems often pick clips that match the words in the request but do not fit the previous clip. For example, they may switch to a different person, a different scene, or a later stage where the action has already happened. This paper studies how to make video retrieval more consistent over time. We introduce a test setting that checks whether a system can choose the correct next clip while keeping both the visual state and the identity of the scene consistent. We also propose CAST, a lightweight method that uses the previous clips and the instruction to predict what the next clip should look like. Across several procedural video datasets, CAST improves the selection of coherent next clips and can also help rerank generated video candidates toward more natural continuations.