SARSteer: Safeguarding Large Audio Language Models via Safe-Ablated Refusal Steering

Large Audio–Language Models (LALMs) are becoming essential as a powerful multimodal backbone for real-world applications. However, recent studies show that audio inputs can more easily elicit harmful responses than text, exposing new risks toward deployment. While safety alignment has made initial advances in LLMs and Large Vision–Language Models (LVLMs), we find that vanilla adaptation of these approaches to LALMs faces two key limitations: 1) LLM-based steering fails under audio input due to the large distributional gap between activations, and 2) prompt-based defenses induce over-refusals on benign-speech queries. To address these challenges, we propose Safe-Ablated Refusal Steering (SARSteer), an effective inference-time defense framework for LALMs. Specifically, SARSteer leverages text-derived refusal steering to enforce rejection without manipulating audio inputs and introduces decomposed safe-space ablation to mitigate over-refusal. Extensive experiments demonstrate that SARSteer significantly improves harmful-query refusal while preserving benign responses, establishing a principled step toward safety alignment in LALMs. The codes and constructed datasets are released at https://github.com/linweiii/SARSteer.

Large audio-language models now power voice assistants and real-time speech interactions. - Problem: When people speak to AI voice assistants like Siri or ChatGPT Voice, it is much easier to trick these systems into saying harmful things compared to typing text. Existing safety defenses either fail with audio inputs or make the AI so cautious that it refuses to answer harmless questions. - Solution: We developed SARSteer, a new defense method that makes audio AI safer without retraining the entire model. Think of it as gently adjusting the steering wheel rather than rebuilding the car. Our method learns what "refusal" looks like in the AI's internal reasoning using text, then applies that knowledge to audio. To prevent over-caution, we use a mathematical technique (principal component analysis) to identify and preserve the AI's ability to recognize safe conversations. - Impact: SARSteer significantly improves the AI's ability to refuse harmful voice queries while keeping its helpful responses to normal questions. This work provides a practical and effective way to make voice AI safer as these systems become more widely used in daily life.