Towards One-to-Many Temporal Grounding
Abstract
Temporal Grounding (TG) aims to localize video segments corresponding to a textual query. Prior research predominantly focuses on single-segment retrieval. Real-world scenarios, however, often require localizing multiple disjoint segments for a single query—a setting we term One-to-Many Temporal Grounding (OMTG). Previous state-of-the-art MLLMs, optimized for one-to-one settings, struggle in this context, often yielding near-zero scores due to a lack of event cardinality perception. To bridge this gap, we present a systematic solution with three key contributions. First, we establish the first comprehensive OMTG benchmark, introducing Count Accuracy (C-Acc) and Effective Temporal F1 (EtF1) as evaluation metrics. Second, we curate a high-quality OMTG dataset comprising 56k samples through a sophisticated construction pipeline. Third, we develop novel temporal and caption reward functions specifically designed for OMTG. In particular, the caption reward leverages Chain-of-Thought reasoning over dense video captions to explicitly guide policy optimization toward both preciseness and completeness. Extensive experiments show our model achieves a new state-of-the-art EtF1 of 43.65\% on OMTG Bench, outperforming Gemini 2.5 Pro and Seed-1.8 by 15.85\% and 15.61\%, respectively.
Lay Summary
When people search through long videos, a single description often corresponds to multiple separate moments. For example, a sports broadcast may contain several goals, or a documentary may revisit the same subject at different times. Yet most existing AI systems are designed to retrieve only one matching clip, causing them to overlook the majority of relevant segments. This work addresses the challenge of locating many disjoint video moments from a single natural-language query. We construct the first large-scale dataset and benchmark for this many-to-one retrieval setting, and develop a training approach that teaches AI models to recognize how many relevant events exist and precisely identify each one. Experiments demonstrate that our method substantially outperforms today's strongest general-purpose AI systems, making video search more complete and reliable for real-world use.