VideoTemp-o3: Harmonizing Temporal Grounding and Video Understanding in Agentic Thinking-with-Videos

In long-video understanding, conventional uniform frame sampling often fails to capture key visual evidence, leading to degraded performance and increased hallucinations. To address this, recent agentic thinking-with-videos paradigms have emerged, adopting a localize–clip–answer pipeline in which the model actively identifies relevant video segments, performs dense sampling within those clips, and then produces answers. However, existing methods remain inefficient, suffer from weak localization, and adhere to rigid workflows. To solve these issues, we propose VideoTemp-o3, a unified agentic thinking-with-videos framework that jointly models video grounding and question answering. VideoTemp-o3 exhibits strong localization capability, supports on-demand clipping, and can refine inaccurate localizations. Specifically, in the supervised fine-tuning stage, we design a unified masking mechanism that encourages exploration while preventing noise. For reinforcement learning, we introduce dedicated rewards to mitigate reward hacking. Besides, from the data perspective, we develop an effective pipeline to construct high-quality long video grounded QA data, along with a corresponding benchmark for systematic evaluation across various video durations. Experimental results demonstrate that our method achieves remarkable performance on both long video understanding and grounding.

Understanding long videos is hard because most systems only look at a few evenly spaced frames, which can easily miss the important moment. We build a new video AI system, VideoTemp-o3, that first looks for the part of the video relevant to a question, then studies that clip more carefully before answering. If its first guess is not good enough, it can try again and narrow down the evidence step by step. We also create a training pipeline that filters and verifies video-question-answer examples, plus a benchmark that tests models on short, medium, long, and very long videos. Experiments show that this approach improves both answering questions and finding the right moments in the video. Our work helps move video AI from guessing based on sparse frames to reasoning over the right evidence, which could improve video search, content analysis, and other long-video applications.