Calibrating Uncertainty for Zero-Shot Adversarial CLIP

CLIP delivers strong zero-shot classification but remains highly vulnerable to adversarial attacks. Prior adversarial fine-tuning work primarily matches predicted logits between clean and adversarial examples, which overlooks uncertainty calibration and may degrade the zero-shot generalization. A common expectation in reliable uncertainty estimation is that predictive uncertainty should increase as inputs become more difficult or shift away from the training distribution. However, we frequently observe the opposite in the adversarial setting: perturbations not only degrade accuracy but also suppress uncertainty, leading to severe miscalibration and over-confidence. This reveals a critical reliability gap beyond robustness. To bridge this gap, we propose an adversarial fine-tuning objective for CLIP considering both accuracy and uncertainty. By reparameterizing CLIP outputs as the concentration parameters of a Dirichlet distribution, we propose a unified representation that captures relative semantic structure and confidence magnitude. This enables holistic distribution alignment under perturbations, moving beyond single-logit anchoring and restoring calibrated uncertainty. Experiments across multiple zero-shot benchmarks demonstrate that our method significantly improves uncertainty calibration and achieves competitive adversarial robustness while preserving clean accuracy.

Modern artificial intelligence systems frequently rely on vision-language models like CLIP to recognize objects in the open world without explicit retraining. However, these systems are highly vulnerable to "adversarial attacks"—subtle, imperceptible pixel manipulations that easily mislead the AI. Crucially, our study reveals a reliability gap beyond mere accuracy: under attack, the AI does not just make mistakes; it becomes blindly over-confident in its wrong answers, suppressing its own uncertainty signals. To bridge this reliability gap, we propose an Uncertainty-Calibrated Adversarial Fine-Tuning (UCAT) objective. By reinterpreting the AI's internal reasoning through a principled mathematical model called the Dirichlet distribution, our method allows the system to evaluate its confidence and surrounding semantic relationships holistically under perturbations. Extensive experiments demonstrate that UCAT successfully restores trustworthy uncertainty logic: when the input becomes corrupted or more difficult, the model naturally expresses higher doubt rather than blind certainty. This advancement provides a critical safeguard for deploying robust and self-aware AI systems in safety-critical, real-world applications.