You Don't Protect if You Don't Expect: Breaking the Key Assumption behind CLIP's Test-Time Defenses

Recent test-time defenses for CLIP claim to preserve zero-shot clean accuracy while improving adversarial robustness. However, we find the reported robustness of six recent proposed state-of-the-art methods substantially overestimated: they fail under basic adaptive attacks. We further observe that these defenses share a common reliance on an indicative measurement that is assumed to capture the distributional difference between clean and adversarial samples and to determine whether the defense should preserve or alter the static model's prediction. We argue that this assumption is the fundamental weakness, and we propose CLIP-MAD (Manipulating Assumed Difference), an adaptive attack strategy designed to break it. CLIP-MAD efficiently expands the adversarial distribution without costly full gradient calculations and can be flexibly combined with existing attack baselines to further boost attack strength. Experiments across 13 datasets demonstrate that CLIP-MAD produces strong adversarial samples that markedly reduce the robustness of diverse test-time defenses, revealing a false sense of security in CLIP’s zero-shot robustness. Code will be available at https://github.com/rzzhang222/CLIP-MAD.

Previous researches show that test-time adversarial defenses for traditional classification can be vulnerable against adaptive attacks (attacks specifically designed for the defense). However, this vulnerability is not explored in CLIP-based zero shot inference yet. This paper explores the false sense of security of test-time defenses for CLIP's zero-shot inference by proposing an efficient and generalizable adaptive attack. By targeting the common indicative measurement step in the pipeline of CLIP's zero-shot test-time defenses, our method has the potential to be easily adapted to newly proposed defenses for vulnerability evaluation and also lowers the information barriers of conducting adaptive attack. We hope the proposed CLIP-MAD can built upon the insights of previous defensive researches and help future researchers achieve truly resilience zero-shot robustness.