DDGA: Dirichlet Distributional Gradient Aggregation for Transferable Vision-Language Adversarial Attacks

Vision-Language Models (VLMs) achieve remarkable performance on multimodal tasks but remain highly vulnerable to adversarial examples, making transferable attacks essential for realistic robustness evaluation. Recent Adversarial Evolution Triangle (AET) methods improve transferability by interpolating over a simplex formed by clean and historical adversarial samples, yet rely on finite random sampling to approximate effective perturbation distributions, which is unstable under limited budgets. In this paper, we propose Dirichlet Distributional Gradient Aggregation (DDGA), a distribution-aware adversarial attack framework that explicitly models and optimizes perturbations over the AET simplex. DDGA parameterizes simplex mixing weights with a learnable Dirichlet policy and optimizes the expected adversarial objective via policy gradient, replacing heuristic sampling with principled distributional optimization. Moreover, we exploit the closed-form covariance of the learned distribution to construct orthogonal perturbations that enhance gradient diversity. Extensive experiments on image-text retrieval and image captioning demonstrate that DDGA consistently outperforms state-of-the-art transfer-based attacks across multiple VLM architectures.

Vision-language models are AI systems that learn to connect visual content with natural language, enabling tasks such as image-text retrieval, image captioning, and visual question answering. Despite strong performance, these models can be sensitive to small, human-imperceptible changes in both images and text. Such changes may cause a model to retrieve the wrong description, generate an inaccurate caption, or otherwise misunderstand the relationship between an image and language. This paper studies how to evaluate these vulnerabilities in a realistic setting, where an attack designed using one model may need to transfer to other, unseen models. We introduce DDGA, a method that improves this evaluation by learning a distribution over useful adversarial changes rather than relying on a small number of random trials. This allows the method to focus on more informative directions while still preserving diversity in the generated perturbations. Experiments on image-text retrieval and image captioning show that DDGA produces adversarial examples that transfer more consistently across different vision-language models than prior methods. These results provide a more effective way to reveal weaknesses in multimodal AI systems, which can help guide the development of models that are more reliable and robust in practice.