Sparse Tokens Suffice: Jailbreaking Audio Language Models via Token-Aware Gradient Optimization

Jailbreak attacks on audio language models (ALMs) optimize audio perturbations to elicit unsafe generations, and they typically update the entire waveform densely throughout optimization. In this work, we investigate the necessity of such dense optimization by analyzing the structure of token-aligned gradients in ALMs. We find that gradient energy is highly non-uniform across audio tokens, indicating that only a small subset of token-aligned audio regions dominates the optimization signal. Motivated by this observation, we propose Token-Aware Gradient Optimization (TAGO), which enables sparse jailbreak optimization by retaining only waveform gradients aligned with audio tokens that have high gradient energy, while masking the remaining gradients at each iteration. Across three ALMs, TAGO outperforms baselines, and substantial sparsification preserves strong attack success rates (e.g., on Qwen3-Omni, $\mathrm{ASR}_{l}$ remains at 86% with a token retention ratio of 0.25, compared to 87% with full token retention). These results demonstrate that dense waveform updates are largely redundant, and we advocate that future audio jailbreak and safety alignment research should further leverage this heterogeneous token-level gradient structure.

Audio language models are AI systems that listen to spoken instructions and answer in natural language. As with text chatbots, people can sometimes craft inputs that make these systems ignore safety rules; these failures are often called jailbreaks. Existing audio jailbreak methods usually modify the whole audio waveform during optimization, but it is unclear whether every part of the audio is actually necessary for changing the model’s behavior. We study the model’s gradients, which indicate where changing the audio is most likely to affect the response, and find that most of this signal is concentrated in a small number of audio-token regions. Based on this finding, we propose Token-Aware Gradient Optimization (TAGO), which updates only the high-signal regions instead of the entire waveform. Across three audio language models, TAGO remains effective even when it keeps only a small fraction of token-aligned regions. This suggests that dense waveform updates are often redundant for audio jailbreak optimization. More importantly, the results reveal a token-level structure that can help researchers better evaluate audio-model safety and design defenses that focus on the parts of audio inputs most responsible for unsafe behavior.