CBV: Clean-label Backdoor Attacks on Vision Language Models via Diffusion Models

Vision-Language Models (VLMs) have achieved remarkable success in tasks such as image captioning and visual question answering (VQA). However, as their applications become increasingly widespread, recent studies have revealed that VLMs are vulnerable to backdoor attacks. Existing backdoor attacks on VLMs primarily rely on data poisoning by adding visual triggers and modifying text labels, where the induced image–text mismatch makes poisoned samples easy to detect. To address this limitation, we propose the Clean-Label Backdoor Attack on VLMs via Diffusion Models (CBV), which leverages diffusion models to generate natural poisoned examples via score matching. Specifically, CBV modifies the score during the reverse generation process of the diffusion model to guide the generation of poisoned samples that contain triggered image features. To further enhance the effectiveness of the attack, we incorporate the textual information of the triggered images as multimodal guidance during generation. Moreover, to enhance stealthiness, we introduce a GradCAM-guided Mask (GM) that restricts modifications to only the most semantically important regions, rather than the entire image. We evaluate our method on MSCOCO and VQA v2 with four representative VLMs, achieving over 80\% ASR while preserving normal functionality.

Vision-language models, which can understand both images and text, are now widely used in applications such as image description and visual question answering. However, these systems can be secretly manipulated through “backdoor attacks,” where hidden malicious behaviors are implanted during training. Existing attacks often modify both images and their text descriptions, creating unnatural mismatches that make the poisoned data easier to detect. In this paper, we propose a new attack method called CBV that creates poisoned training examples in a more natural and difficult-to-detect way. Instead of directly inserting obvious patterns into images, our method uses image generation models to subtly alter images while preserving their original appearance and meaning. We further guide the generation process using both image and text information to make the attack more effective. In addition, we only modify the most visually important regions of an image, which helps keep the changes less noticeable. Experiments on two widely used benchmark datasets and four representative vision-language models show that our method can successfully trigger malicious behaviors with a high success rate while maintaining the models’ normal performance on clean inputs.