MetaPerch: Learning from metadata for bioacoustics foundation models

Bioacoustic foundation models rely on large-scale citizen science platforms like Xeno-Canto for geographically and ecologically diverse data. Recent work has shown that supervision alone can produce SotA species detection models when trained on this large-scale data---however, there remains unutilized potential in the form of recording metadata readily available within these community-driven data hubs. In this work, we explore the use of metadata---such as location and time---as auxiliary supervision signals, allowing the model to leverage species-metadata correlations in its learned representation. Auxiliary metadata losses provide additional information beyond vocalizations alone that can encourage a richer, more robust representation that generalizes better to species distribution and acoustic domain shifts---important challenges for deployment in real-world passive acoustic monitoring (PAM) settings. We introduce MetaPerch, a new foundation model that achieves strong species identification performance across multiple challenging domains and present an extensive empirical study of the effects of 9 diverse metadata sources on 17 bioacoustic datasets.

In order to build a broadly useful model to process and understand sounds from nature ("bioacoustics", the study of biological sounds), scientists and engineers use data collected via "citizen-science" and crowdsourced platforms. These publicly accessible repositories serve as a place where nature enthusiasts can upload audio data with tags that describe the dominant ("foreground") species vocalizing in the audio clips. Beyond the raw audio, additional tags, called "metadata", are sometimes added, including additional animals heard in the background; date and time of recording; where the recording was taken; recording device used; etc. The best models available for detecting and classifying which species are present in audio have historically been built only using raw audio data---but we argue that these other sources of metadata can be very useful in developing more robust, globally useful models (particularly for certain challenging environmental conditions or species). We build a new model---MetaPerch---and show that it improves performance over our baseline and is competitive compared to the current "state-of-the-art" on a large number of highly varied bioacoustic tasks. We present both a new model that practitioners can use, as well as an extensive empirical study on how to use this metadata in building new, broadly useful models. We hope that this work further motivates citizen scientists and practitioners to include metadata when collecting crowdsourced bioacoustic data.