LALM-as-a-Judge: Benchmarking Large Audio-Language Models for Safety Evaluation in Multi-Turn Spoken Dialogues

Evaluation of socially unsafe content in spoken dialogues remains text-centric, missing prosody and transcription failures. We present LALM-as-a-Judge, which includes an open benchmark of 24,000 multi-turn spoken dialogues with one localized unsafe turn, generated out of 8 socially unsafe categories and 5 severity levels. We evaluate 6 large audio-language models (LALMs) as judges, open and closed-source, in text-only, audio-only, and multimodal setups by their sensitivity, severity-order specificity, and turn-position bias for socially harmful content in the dialogue. Results show that audio contributes non-lexical evidence beyond transcript semantics and that multimodal gains are not universal but can be text-anchored, balanced, conservative, and interfering, which we link to the audio pathway bottlenecks and fusion limits. We position the benchmark as diagnostic and derive practitioner guidance for model, modality, and prompts choices.

Voice assistants are starting to handle tasks like tutoring, customer support, and everyday conversation, so they need reliable ways to notice harmful exchanges. Today, many safety checks first turn speech into text and then judge the words, which can miss tone, emphasis, or speech-recognition mistakes. We built a new testbed for this problem: 24,000 spoken multi-turn conversations where one turn is carefully changed to include an unsafe moment at different levels of seriousness, while the rest of the conversation stays the same. Human reviewers confirmed that these changed conversations were usually recognized as unsafe and that the seriousness labels made sense. We then tested several open audio-language models and a text-only language model to see how well they could score dialogue safety from audio, text, or both. We found that there is no single best setup: text can be strong when transcripts are accurate, audio can help when how something is said matters, and combining audio with text can sometimes make judgments less stable. Even small transcription errors can cause safety judges to miss milder problems. Our benchmark and recommendations can help researchers and developers choose safer evaluation pipelines for future voice agents.