Neural-Inspired Modeling of Auditory Selection and Compensation for Audio-Visual Speech Separation

Current audio-visual speech separation (AVSS) models typically rely on implicit multimodal fusion, but the absence of explicit modality alignment and reliability modeling often causes semantic misalignment and contaminates speech representations. The brain addresses this with a hierarchy: top-down auditory selection uses visual priors to maintain target-consistent acoustics, while bottom-up cross-modal compensation integrates temporally aligned articulatory cues to reconstruct and stabilize speech. Guided by this principle, we present Neuro-SCNet, an AVSS architecture that makes selection and compensation explicit and reliability-aware. The Auditory Selection Mechanism applies top-down, visually guided gain along the audio pathway to isolate target time-frequency units and suppress distractors. The module preserves the auditory trace with an identity bypass and adds controlled visual refinements via a residual path. A synchrony-driven gate reduces the influence of low-confidence visual cues. Additionally, a lightweight pre-alignment for visual feature pre-processing estimates and corrects small temporal offsets, and a compact magnitude-phase encoder is used to preserve fine acoustic detail to stabilize reconstruction. Evaluations on LRS2, LRS3, and VoxCeleb2 show state-of-the-art separation with improved efficiency, supporting the value of explicit selection and reliability-aware compensation.

Imagine trying to follow one person’s voice in a crowded room while watching their face. Computers can also use both sound and lip movement to separate a target speaker from other voices, but many existing systems combine these cues too early, so misleading sound or video information can hurt the result. Our work builds a speech-separation system inspired by how people listen. It first uses the speaker’s visible mouth movements to help choose the parts of the sound that likely belong to the target speaker. It then uses reliable visual cues to fill in missing or weakened speech information, while keeping the original audio as a safe reference. The system also checks whether the video and audio are well matched in time, and relies less on video when it may be unreliable. Tests on several speech datasets show that this approach produces clearer separated speech with a relatively compact model. This can support applications such as online meetings, hearing assistance, and speech technologies used in noisy environments.