DART: Distribution-Aware Adaptive Relational Transfer for Adversarial Attacks against Closed-Source MLLMs

This paper studies the critical problem of targeted adversarial attacks against closed-source MLLMs, which aim to generate highly transferable adversarial samples with open-source MLLMs. Previous approaches typically focus on maximizing the similarity of latent representations between adversarial samples and target samples. However, these approaches could overfit specific target samples with severely limited generalization ability to closed-source MLLMs. Towards this end, we propose a novel approach named Distribution-aware Adaptive Relational Transfer (DART) for adversarial attacks against closed-source MLLMs. The core of our DART is to adopt a statistical lens to characterize the intrinsic semantics of images for more generalized and robust alignment. In particular, each augmented image is considered an example from the intrinsic distribution of the original image. Then, we utilize non-parametric Energy Distance to measure the distribution divergence, which is naturally adopted for the semantic alignment in the hidden space. To further enhance transferability to specific target models, we learn a graph neural network (GNN) to explore the complex relations between source and target MLLMs on transferability and adaptively select surrogate models to maximize transferability across diverse targets. Extensive experiments on benchmark datasets validate the superior robustness and effectiveness of the proposed DART in comparison to various competing baselines.

Modern AI systems such as ChatGPT, Gemini, and Claude can analyze both images and text, and they are now widely deployed in real-world applications. However, it remains unclear how easily these systems can be fooled by tiny, deliberate changes to a picture. These changes are invisible to humans, yet enough to make the AI describe the image incorrectly. Most existing evaluation methods only work on small academic models and fail to generalize to today's commercial AI systems, which operate as inaccessible black boxes. We propose DART, a method that crafts subtle image modifications capable of consistently deceiving these commercial AI systems. Two key ideas drive its effectiveness. First, instead of imitating only one feature of a target image, we align the broader pattern of meaning shared across many slightly different views of that image. This makes the attack far more robust to differences between AI systems. Second, we train a "router" network that automatically picks, for each new task, the best combination of open-source AI models to mimic during the attack. By revealing how commercial AI can be tricked, our work helps developers identify and patch these weaknesses before they lead to real-world harm.