Two Modalities Are Better Than One: Efficient Adversarial Purification via Multimodal Diffusion Models

Adversarial purification uses generative models to restore clean data distributions from unseen attacks without retraining classifiers. However, unimodal diffusion-based approaches struggle to preserve semantic consistency, while recent multimodal variants rely on computationally expensive adversarial training or distillation. Both approaches often lack theoretical guarantees. In this work, we propose MultiDAP, a novel framework leveraging multimodal diffusion models for efficient adversarial purification. MultiDAP first learns continuous class-agnostic prompts from clean data to capture rich semantic priors, replacing rigid hand-crafted templates. Guided by these prompts, MultiDAP purifies adversarial inputs by minimizing a regularized DDPM loss for only a few steps (e.g., 5-20). We provide theoretical guarantees for both the likelihood improvement via prompt learning and the convergence of the purification process. Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet-1K demonstrate that MultiDAP matches the robustness of state-of-the-art baselines but with improved efficiency.

This work is on how to remove malicious and human unrecognizable perturbations or noises in images. These malicious and human imperceptible perturbations or noises can mislead the classification models to wrongly identify the images as another class, for example, classify a dog into a car. The nature of this noising removal, i.e., denoising, is to find the true semantic label of the images. Here we propose to use diffusion models with both visual and semantic modalities, i.e., multimodal diffusion models, to complete this denoising task, where semantic inputs are learned from unperturbed clean images. Because multimodal diffusion models have strong powers to generate images and different modalities in multimodal diffusion models have high alignment, we can successfully remove these perturbations or noises. Theoretical studies and numerical experiments both demonstrate this success.