FrameOracle: Learning What to See and How Much to See in Videos

Vision-language models (VLMs) advance video understanding but operate under tight computational budgets, making performance dependent on selecting a small, high-quality subset of frames. Existing frame sampling strategies, such as uniform or fixed-budget selection, fail to adapt to variations in content density or task complexity. To address this, we present FrameOracle, a lightweight, plug-and-play module that predicts both (1) which frames are most relevant to a given query and (2) how many frames are needed. FrameOracle is trained via a curriculum that progresses from weak proxy signals, such as cross-modal similarity, to stronger supervision with FrameOracle-41K, the first large-scale VideoQA dataset with validated keyframe annotations specifying minimal sufficient frames per question. Extensive experiments across five VLMs and six benchmarks show that FrameOracle reduces 16-frame inputs to an average of 10.4 frames without accuracy loss. When starting from 64-frame candidates, it reduces inputs to 13.9 frames on average while improving accuracy by 1.5%, achieving state-of-the-art efficiency–accuracy trade-offs for scalable video understanding.

Many AI systems can answer questions about videos, but they often cannot watch every frame because long videos are expensive to process. A common shortcut is to sample frames evenly or always keep the same number of frames, but this can miss important moments in some videos and waste computation on unhelpful frames in others. We introduce FrameOracle, a small plug-and-play module that helps an existing video AI system decide which frames are useful for a specific question and how many frames it should keep. To train it, we also build FrameOracle-41K, a dataset that shows, for each video question, the minimal set of frames needed to answer correctly. Across several video understanding benchmarks and different AI models, FrameOracle reduces the number of frames the model needs to process while preserving or improving accuracy. This makes video understanding more efficient and scalable, especially for long videos where most frames may not be necessary for answering a given question.