Poster

OmniDenseCap: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions

Linli Yao ⋅ Yuancheng Wei ⋅ Yaojie Zhang ⋅ Lei Li ⋅ Xinlong Chen ⋅ Feifan Song ⋅ Ziyue Wang ⋅ Kun Ouyang ⋅ Yuanxin Liu ⋅ Lingpeng Kong ⋅ Qi Liu ⋅ Pengfei Wan ⋅ Kun Gai ⋅ Yuanxing Zhang ⋅ Xu SUN

Abstract

This paper proposes Omni Dense Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps. To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions, enabling readers to vividly imagine the video content scene-by-scene, akin to a cinematographic screenplay. To facilitate research, we construct OmniDCBench, a high-quality human-annotated benchmark, and propose SodaM, a unified metric that evaluates time-aware detailed descriptions while mitigating scene boundary ambiguity. Furthermore, we construct a training dataset OmniDenseCap-40K and present Omni-Captioner-7B, a strong baseline trained via SFT and GRPO with task-specific rewards. Extensive experiments demonstrate that Omni-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro, while its generated dense descriptions significantly boost downstream capabilities in audio-visual reasoning (DailyOmni and WorldSense) and temporal grounding (Charades-STA). All datasets, models, and code will be made publicly available.