Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox

Audio large language models (Audio LLMs) demonstrate strong performance on speech understanding tasks, yet their ability to understand paralinguistic information remains limited. To systematically quantify this issue, we introduce VoxParadox, an adversarial benchmark with 2,000 verified examples, spanning 10 paralinguistic tasks, created with controlled speech synthesis to intentionally mismatch transcript claims and speaking style, enabling direct measurement of speech paralinguistic understanding. Evaluation of a diverse set of Audio LLMs reveals consistently low accuracy on acoustic ground truth and a strong tendency to follow language-implied (incorrect) answers. To understand the cause of this gap, we perform layer-wise probing and find that (i) paralinguistic cues can degrade in deeper encoder layers and at the encoder--LLM interface, and (ii) even when such cues are available in audio tokens, the language model frequently ignores them. To address these problems, we propose Prompt-Conditioned Layer Mixer (PCLM), which adaptively combines information from multiple audio layers based on the input prompt, and pair it with Direct Preference Optimization (DPO) to explicitly prefer acoustically supported options over language-implied alternatives. These methods substantially improve Audio LLM paralinguistic understanding, improving Audio Flamingo 3 from 17.40% to 65.20% on VoxParadox, and from 37.74% to 54.78% on MMSU paralinguistic subset. Our project page is available at https://voxparadox.github.io/.

Beyond words, the human voice carries rich information—emotion, age, gender, pitch, and speaker identity. Understanding these vocal cues is essential for healthcare, accessibility, and human-computer interaction. Today's AI speech systems pair audio processing with powerful language models, but it remains unclear whether they truly listen to vocal cues or simply rely on the words being said. We built VoxParadox, a test set of 2,000 speech clips where the words deliberately contradict the voice—for example, an elderly speaker saying "I am a child." Testing 12 leading speech AI systems, we found most followed the misleading words, choosing transcript-implied answers 64% of the time while only 15% matched the actual voice. Surprisingly, probing inside these models revealed the vocal information was already there—they simply never learned to use it. Our method unlocks this trapped knowledge: it gives models adaptive access to acoustic details they already possess but would otherwise ignore, and trains them to prefer voice-based evidence over misleading words. This improved accuracy from as low as 17% to 65–72%. Our benchmark and methods offer a step toward speech AI that understands not just what we say, but how we say it.