SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models

Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but remain highly fragile under adversarial perturbations. Recent test-time adaptation defenses improve robustness by leveraging many augmented views, but this leads to impractical slowdown and a clear robustness-throughput trade-off. To address this challenge, we present Stability and Suitability-guided Test-time Prompt Tuning (SS-TPT), evaluating the quality of each augmented view via two complementary scores: (1) stability, measuring prediction invariance to weak augmentations, and (2) suitability, measuring feature-space density among views. These stability and suitability (SS) scores guide both adaptation and inference through an SS-guided consistency loss and an SS-weighted prediction, amplifying trustworthy views while suppressing corrupted ones. Extensive experiments demonstrate that SS-TPT significantly outperforms prior state-of-the-art methods, achieving superior robustness-throughput trade-offs across diverse datasets and varying numbers of views, thereby demonstrating both strong practicality and generality. Our code is available at https://github.com/sunoh-kim/SS-TPT.

Modern vision-language AI systems can recognize images using natural language descriptions, but they can be surprisingly fragile: tiny changes to an image, often invisible to humans, can cause them to make wrong predictions. Existing defenses can make these systems safer, but they often require the model to examine many altered versions of the same image, which slows it down and makes real-world use less practical. We propose a method that helps the model decide which altered versions of an image are trustworthy. Instead of treating all versions equally, our method gives more importance to versions whose predictions remain steady under small changes and whose visual patterns agree with other versions of the same image. The model then learns from these more reliable views and relies on them when making its final decision. This makes vision-language models more robust against misleading image changes while keeping them efficient. Across many image recognition benchmarks, our method improves reliability compared with previous defenses and works well even when only a small number of image versions are available.