AudioMosaic: Contrastive Masked Audio Representation Learning

Audio self-supervised learning (SSL) aims to learn general-purpose representations from large-scale unlabeled audio data and has achieved remarkable progress in recent years. While most prior work relies on generative reconstruction objectives, contrastive approaches remain relatively underexplored, in part due to the high computational cost of designing effective augmentation strategies and the large batch sizes typically required for pre-training. In this work, we introduce AudioMosaic, an audio encoder for general audio understanding. During pre-training, AudioMosaic applies time–frequency masking to spectrogram patches to form paired inputs, employing an elegant and efficient augmentation strategy that significantly reduces computational cost while supporting large-batch training. The AudioMosaic encoder learns discriminative utterance-level representations that exhibit strong transferability across datasets, domains, and acoustic conditions. Extensive experiments demonstrate that AudioMosaic achieves state-of-the-art performance on multiple standard benchmarks. Moreover, we show that the pretrained AudioMosaic encoder enhances audio perception when integrated with large language models (LLMs).

Audio is everywhere, from speech assistants and environmental monitoring to tools that detect synthetic or manipulated sounds. To make these systems work well, AI models need to learn useful patterns from large amounts of audio, even when the audio has no human-provided labels. In this paper, we introduce AudioMosaic, a new way to train an audio AI model by showing it two incomplete but complementary versions of the same sound and asking it to recognize that they belong together. The key idea is to hide different parts of an audio spectrogram, which is a visual representation of sound over time and frequency. By comparing these partially hidden views, the model learns the broader structure of a sound rather than relying only on small local details. This makes the learned audio features more useful across many tasks and conditions. AudioMosaic performs strongly on standard audio benchmarks, including sound classification, speech-related tasks, and environmental sound deepfake detection. It also improves audio-language models, helping them better connect sounds with text. Overall, this work provides an efficient and effective approach for building general-purpose audio understanding systems.