MADA-Attack: Transferable Multi-modal Attention Distraction Adversarial Attack against Vision Language Models

Vision Language Models (VLMs) achieve strong performance across multi-modal tasks but remain vulnerable to universal adversarial perturbations (UAPs). Existing UAP methods mainly operate on the visual modality, overlooking structured textual semantics and cross-modal interactions, which limits their ability to disrupt alignment and generalize across tasks and model architectures. To address these limits, we propose Multi-modal Attention Distraction Adversarial Attack (MADA-Attack) framework. We begin by conducting several insight experiments and discover that modality attention distributes differently over layers and early phase of optimization is decisive. Building on these observations, we introduce Semantic Token Manipulation (STM) to steer text-guided attention, and Fused Embedding Training (FET) to jointly optimize textual and visual embedding losses for coordinated misalignment. We further incorporate an Adaptive Data Augmentation (ADA) strategy that dynamically balances attack strength, transferability, and training efficiency. Extensive experiments demonstrate that MADA-Attack consistently achieves state-of-the-art performance and strong transferability while remaining computationally lightweight, with an average ASR of 82.60\% and 73.42\% in zero-shot classification and image captioning tasks. For the visual question answering (VQA) and I-T Retrieval task, our method exceeds the SOTA baseline by 10\%. Our code is available at this GitHub Repository.

Vision-language AI systems can describe images, answer visual questions, and match images with text. Their success comes from learning a shared understanding between language and vision. However, this ability also creates new security risks. In this work, we study universal adversarial attacks. In these attacks, a single small and almost invisible perturbation can mislead an AI system across many images and tasks. Most existing methods only manipulate visual information. They treat the problem as an image-only attack. However, modern vision-language systems also depend strongly on how images and text are connected inside the model. Based on this observation, we propose MADA-Attack, a new attack framework that targets the interaction between language and vision. Instead of only changing image features, our method also uses language-guided signals to distract the model’s internal attention. This weakens the connection between visual and textual information and improves attack transferability across different models and tasks. Our results show that cross-modal alignment is an important but underexplored vulnerability in modern AI systems.