LightAVSeg: Lightweight Audio-Visual Segmentation

Audio-Visual Segmentation (AVS) targets pixel level localization of sounding emitting objects in videos. However, existing models rely on dense cross-modal attention with quadratic computational cost, limiting their suitability for resource efficient deployment. Most efficiency oriented methods focus on backbone reduction and overlook the interaction module as the primary bottleneck. This paper proposes LightAVSeg, a lightweight framework that replaces heavy attention with a decoupled design for semantic filtering and spatial grounding, resulting in interaction costs that scale linearly with spatial resolution. Furthermore, we introduce an auxiliary alignment loss to enforce semantic consistency during training with zero inference overhead. Extensive experiments demonstrate that LightAVSeg achieves a new state-of-the-art among lightweight methods: with 20.5M parameters (~1/7 of AVSegFormer), it reaches 50.4 mIoU on the MS3 benchmark and enables efficient inference on a mobile processor.

Many videos contain several visible objects, but only some of them are actually making sound. Audio-visual segmentation aims to find the exact image regions of the sounding objects, such as a barking dog or a playing instrument. Existing methods can be accurate, but they often rely on heavy sound-image matching, which makes them slow and difficult to use on mobile devices or headsets. We propose LightAVSeg, a lightweight model designed to solve this problem efficiently. Our key idea is to separate the task into two simpler steps: using audio to identify what object is likely producing sound, and using visual features to locate where it appears. We also add a training signal that helps the model align sound with the correct visual regions, without increasing inference cost. LightAVSeg is smaller, faster, and accurate, making audio-guided video understanding more practical for real-world on-device applications.