VideoBrain: Learning Adaptive Frame Sampling for Long Video Understanding

Long-form video understanding remains challenging for Vision-Language Models (VLMs) due to the inherent tension between computational constraints and the need to capture information distributed across thousands of frames. Existing approaches either sample frames uniformly (risking information loss) or select keyframes in a single pass (with no recovery from poor choices). We propose VideoBrain, an end-to-end framework that enables VLMs to adaptively acquire visual information through learned sampling policies. Our approach features dual complementary agents: a CLIP-based agent for semantic retrieval across the video and a Uniform agent for dense temporal sampling within intervals. Unlike prior agent-based methods that rely on text-only LLMs orchestrating visual tools, our VLM directly perceives frames and reasons about information sufficiency. To prevent models from invoking agents indiscriminately to maximize rewards, we introduce a behavior-aware reward function coupled with a data classification pipeline that teaches the model when agent invocation is genuinely beneficial. Experiments on four long video benchmarks demonstrate that VideoBrain achieves +3.5\% to +9.0\% improvement over the baseline while using 30-40\% fewer frames, with strong cross-dataset generalization to short video benchmarks. The code is available at https://github.com/junbo-zou/VideoBrain.

How can an AI answer questions about a two-hour video when it can only look at a small number of frames? Choosing the wrong frames means missing critical information — with no way to recover. VideoBrain solves this by teaching an AI to reason about whether it has seen enough, then decide whether to look for more. It uses two tools: one that searches the entire video for visually relevant scenes, and one that samples a specific time interval for sequential details. Crucially, the model also learns when not to use these tools — avoiding unnecessary searches when the initial frames already contain the answer. On four challenging long-video benchmarks, VideoBrain improves accuracy by 3.5–9.0% while using 30–40% fewer frames than comparable models. It also generalizes to short videos, showing that smarter, adaptive viewing beats simply looking at more frames.