Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models

Vision-Language Models (VLMs), such as CLIP, have achieved significant zero-shot performance on downstream tasks with various fine-tuning adaptation methods. However, recent studies have proven that adversarial attacks can significantly degrade the inference ability of VLMs, posing substantial risks to their practical applications. Prevalent test-time adaptation methods typically rely on multi-view augmentation to implement various fine-tuning strategies, which struggle to identify semantic information and are prone to destroying discriminative regions in fine-grained scenarios. To address these limitations, we propose Attention-Guided Test-Time Prompt Tuning (A-TPT), a semantics-preserving method designed for test-time adaptation. We first refine the gradient attention rollout mechanism to identify semantically meaningful regions surviving under adversarial attacks. Furthermore, we leverage them to guide the spatially varying augmentation intensities and multi-view ensemble for prompt tuning and inference. Extensive experiments demonstrate that A-TPT outperforms existing test-time adaptation methods on both adversarial and clean data. Codes are available at https://github.com/SEU-VIPGroup/A-TPT.

Vision-language models, such as CLIP, can recognize images by connecting visual content with textual descriptions, and they have shown strong performance on many tasks without requiring large amounts of task-specific training data. However, these models can be easily misled by adversarial attacks, where small and often imperceptible changes to an image cause the model to make incorrect predictions. This problem is especially serious in fine-grained recognition tasks, such as distinguishing between similar bird species, pet breeds, or aircraft models, where the correct decision often depends on subtle visual details. In this work, we propose Attention-Guided Test-Time Prompt Tuning (A-TPT), a method that improves the reliability of vision-language models during inference. Instead of treating all image regions equally, A-TPT helps the model identify and preserve meaningful regions that remain useful even when the input image is attacked. It then uses this information to guide image augmentation, prompt tuning, and prediction aggregation. Experiments show that A-TPT improves model performance on both adversarially attacked images and clean images, making vision-language models more robust and practical for real-world applications.