VocSim A Training-free Benchmark for Zero-shot Content Identity in Single-source Audio

General-purpose audio representations aim to map acoustically variable instances of the same event to nearby points, resolving content identity in a zero-shot setting. Unlike supervised classification benchmarks that measure adaptability via parameter updates, we introduce VocSim, a training-free benchmark probing the intrinsic geometric alignment of frozen embeddings, with no parameters updated and no labels used (a label-free PCA whitening is fit per subset to correct anisotropy). VocSim aggregates 125k single-source clips from 19 corpora spanning human speech, animal vocalizations, and environmental sounds, isolating content representation from source separation (polyphonic mixtures are out of scope). We evaluate embeddings with Precision@k for local purity and the Global Separation Rate (GSR) for point-wise class separation, calibrated by lift over an empirical permutation baseline. A simple pipeline of frozen Whisper features, time–frequency pooling, and label-free PCA yields strong zero-shot performance with stable GSR rankings across domains (Kendall's τ = 0.60). However, on blind low-resource speech (Shipibo-Conibo, Chintang), local retrieval collapses while remaining above chance, exposing a cross-lingual speech generalization gap. As external validation, our top embeddings predict avian perceptual similarity, improve bioacoustic classification, and achieve state-of-the-art on the HEAR benchmark. We release data, code, and a public leaderboard.

When you hear someone say the same word twice, once loudly, once whispered, once on the phone, you instantly recognize it as the same word. Modern AI audio models are supposed to do the same: ignore surface differences and focus on what the sound is. But evaluating whether they actually do this is hard, because today's benchmarks mostly measure how well a model can be re-trained for a task, not how well its built-in understanding of sound is organized. We introduce VocSim, a benchmark that takes a frozen, off-the-shelf AI audio model and asks a simple geometric question: do clips of the same thing, the same word, the same bird call, the same dog bark, end up near each other in the model's internal representation, without any retraining? We test this on 125,000 short audio clips drawn from 19 collections of human speech, animal vocalizations, and environmental sounds. Our most striking finding is a cross-lingual gap: even the best current models, which work well on English and other high-resource languages, almost completely fall apart when asked to organize the sounds of low-resource languages like Shipibo-Conibo and Chintang. They can still tell these sounds apart from random noise, but they cannot reliably group them by content. This is a concrete, measurable form of bias: audio AI today is fluent in some of the world's languages and effectively deaf in others. On the positive side, the same lightweight pipeline that scores best on VocSim, using features from OpenAI's Whisper model, also predicts what zebra finches perceive as similar songs, helps classify mouse vocalizations, and achieves state-of-the-art results on an existing audio benchmark (HEAR). This suggests that the internal geometric structure we measure is a useful, general-purpose proxy for how well an audio model will perform on many downstream tasks. We release the data, code, and a public leaderboard so others can test their models.