Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Models

Pre-trained vision-language models (VLMs) such as CLIP have demonstrated strong zero-shot capabilities across diverse domains, yet remain highly vulnerable to adversarial perturbations that disrupt image-text alignment and compromise reliability. Existing defenses typically rely on adversarial fine-tuning with labeled data, limiting their applicability in zero-shot settings. In this work, we identify two key weaknesses of current CLIP adversarial attacks—lack of semantic guidance and vulnerability to view variations—collectively termed semantic and viewpoint fragility. To address these challenges, we propose Self-Calibrated Consistency (SCC), an effective test-time defense. SCC consists of two complementary modules: Semantic consistency, which leverages soft pseudo-labels from counterattack warm-up and multi-view predictions to regularize cross-modal alignment and separate the target embedding from confusable negatives; and Spatial consistency, aligning perturbed visual predictions via augmented views to stabilize inference under adversarial perturbations. Together, these modules form a plug-and-play inference strategy. Extensive experiments on 22 benchmarks under diverse attack settings show that SCC consistently improves the zero-shot robustness of CLIP while maintaining accuracy, and can be seamlessly integrated with other VLMs for further gains. These findings highlight the great potential of establishing an adversarially robust paradigm from CLIP, with implications extending to broader VLMs such as BioMedCLIP.

Modern AI can recognize what is in a photograph by matching it to short text descriptions — the technology behind photo search, content moderation, and even medical image triage. A serious weakness is that an attacker can add a tiny amount of carefully chosen "noise" to a picture, invisible to the human eye, and trick the system into the wrong answer: a stop sign read as a speed-limit sign, or a medical scan steered to the wrong diagnosis. The usual remedy is to retrain the AI on attacked images, which is expensive and often hurts its everyday accuracy. We propose a lightweight fix called Self-Calibrated Consistency (SCC). Instead of retraining, SCC works only at the moment a question is asked. It runs two simple self-checks: whether the picture still broadly agrees with the kinds of labels the AI was leaning toward, and whether small natural variations of the image (a flip, a touch of blur) all tell the same story. Together these checks let the system ignore the attacker's noise and recover the true meaning. Across 22 standard benchmarks, including everyday photos and medical scans, SCC consistently makes such systems far harder to fool, without losing accuracy on clean images and without any retraining — offering a general, plug-in safety layer for image-and-text AI.