AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes

Audio-visual speaker tracking aims to localize and track active speakers by leveraging auditory and visual cues, enabling fine-grained, human-centric scene understanding. This capability is essential for real-world applications such as intelligent video editing, surveillance, and human–computer interaction. However, existing datasets are largely limited to simple or homogeneous audio-visual scenes with coarse annotations. Such oversimplified settings bias evaluation toward static audio–visual co-occurrence, rather than rigorously assessing robust spatiotemporal modeling and cross-modal reasoning in complex, dynamic scenes. To address these limitations, we introduce AVTrack, a human-centric audio-visual instance segmentation (AVIS) dataset designed for dynamic real-world scenarios. AVTrack features diverse and challenging conditions, including camera motion, visual occlusions, and position changes. Evaluations of representative AVIS methods on AVTrack reveal substantial performance degradation, establishing AVTrack as a challenging benchmark for robust human-centric audio-visual scene understanding in complex environments. We further provide a simple yet effective baseline to facilitate future research. Project website: https://FudanCVL.github.io/AVTrack/

When humans watch a conversation, they effortlessly identify the active speaker by integrating visual and auditory cues. Enabling computers to perform the same task, with applications in automatic video editing, surveillance, and human--computer interaction, has proven considerably more difficult than it appears. A key reason is that existing benchmarks are overly simplistic: speakers remain stationary, cameras do not move, and no occlusion occurs. Consequently, methods that perform well on such clean scenes often fail on cluttered real-world footage. To address this limitation, we introduce AVTrack, a new dataset and benchmark designed to capture the complexity of real-world conversations, featuring dynamic camera motion, mutual occlusion, and varying speaker positions, with every speaker precisely annotated in each frame. When state-of-the-art methods are evaluated on AVTrack, a noticeable drop in accuracy is observed, indicating that current benchmarks may not fully reflect real-world difficulty. AVTrack therefore offers a complementary benchmark for more comprehensive evaluation, accompanied by a simple baseline to facilitate further research.