Video-in-the-Loop: Span-Grounded Long Video QA with Interleaved Reasoning

We present $\textit{Video-in-the-Loop}$ (ViTL), a two-stage long-video QA framework that preserves a fixed token budget by first $\textit{localizing}$ question-relevant interval(s) with a low-fps skim and then $\textit{answering}$ via span-aware reallocation of visual tokens at higher effective frame rate, emitting an interleaved output with both spans and the final option for direct attribution. We also introduce $\textit{VGrounding-QA}$, which converts description based event graphs into $\textit{span-grounded}$ multiple-choice QA by pairing each question with $\textit{ground-truth}$ time span(s) and related reasoning. ViTL is trained end-to-end with an interleaved group-relative objective that couples temporal IoU for localization with answer correctness, allowing credit to flow from answers back to spans without increasing compute. Under fixed token budgets, ViTL attains up to 8.6\% with 50\% less frame input on long-video QA and temporal grounding (e.g., Charades-STA, ActivityNet-Captions) and ablations show that span-aware token reallocation consistently surpasses uniform sampling. Together, $\textit{VGrounding-QA}$ and ViTL provide an interpretable, compute-efficient recipe for scalable long-video QA.

Long videos often contain only a few moments that are needed to answer a question, but today’s video AI systems usually sample frames uniformly and waste much of their limited budget on irrelevant content. This makes long-video question answering less accurate and harder to verify, because the model may give an answer without clearly showing where in the video the evidence came from. We introduce Video-in-the-Loop (ViTL), a two-stage method that first quickly skims the whole video to find the relevant time spans, then zooms into those spans to answer the question using more detailed visual information. To train this behavior, we also build VGrounding-QA, a dataset that pairs each question with the exact video segment or segments needed to answer it. This approach makes video question answering more accurate, interpretable, and compute-efficient. Instead of spending effort on the entire video equally, ViTL focuses computation on the evidence, improving long-video understanding while making the model’s answers easier to check.