Robustifying Vision-Language Models via Test-Time Prompt Adaptation

Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations. Existing test-time adaptation methods typically rely on sample-level confidence heuristics, overlooking the intrinsic distributional structure of the data. This sample-centric approach limits robustness, as it fails to distinguish confident adversarial mispredictions from true semantic consistency. In this work, we observe that adversarial distortion is structurally brittle: while holistic representations are corrupted, semantic integrity is often preserved in the distribution of augmented views. Motivated by this insight, we propose $\texttt{RITA}$, a $\textbf{R}$obust test-t$\textbf{I}$me promp$\textbf{T}$ $\textbf{A}$daptation framework that shifts from sample-level estimates to distribution-level alignment. Specifically, $\texttt{RITA}$ employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment. Furthermore, we introduce a dynamic cache to progressively accumulate reliable cues from the test stream for online refinement. Extensive experiments demonstrate that $\texttt{RITA}$ significantly improves adversarial robustness without compromising clean accuracy.

Vision-language models such as CLIP can recognize images using natural language descriptions, but they can be easily fooled by small, carefully designed changes to an image that are almost invisible to humans. This paper proposes RITA, a method that makes these models more reliable at test time without retraining them or requiring labeled data. Instead of relying on a single view of an image, RITA looks at multiple transformed versions of the same image and uses their shared information to better match the image with the correct text description. It also stores reliable information from previous test examples to further improve future predictions. Experiments on many image classification datasets show that RITA makes vision-language models much more robust to adversarial attacks while keeping strong performance on normal images.