USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning

Test-time adaptation (TTA) has emerged as a popular paradigm for improving the performance of vision–language models (e.g., CLIP) on downstream tasks. Among existing CLIP-based TTA methods, Test-Time Prompt Tuning (TPT) is a pioneering work that optimizes textual prompts using multiple test-time augmentations and remains a strong baseline to date. In this work, we revisit TPT and reveal that its optimization can be interpreted as implicitly learning from self-generated pseudo labels. Building on this perspective, we propose a unified self-ensembling framework (USE) that ensures consistency between the optimization and inference stages. During optimization, we introduce a simple yet effective self-ensembling (SE) strategy that emphasizes the test image itself over its augmented views adaptively to obtain more reliable pseudo labels. To fully exploit the potential of augmentations, we further apply the same strategy at inference time, unifying the objectives of both stages. Notably, SE can also act as a lightweight optimization-free TTA method. Extensive experiments across multiple datasets demonstrate that SE and USE outperform their counterparts, respectively. Furthermore, SE yields consistent performance gains when integrated with existing TTA methods. The code is available at https://github.com/sirujiang/USE.

While AI models that process both images and text are incredibly smart, they can still get confused by unfamiliar images—like mistaking a dog for a cat. We wanted to find a way to help them quickly adapt to new images on the fly, without needing extra clues. Currently, these models try to adapt by teaching themselves. They do this by guessing answers based on altered versions of a test image. The problem is that they often ignore the most reliable piece of evidence: the original image itself. By fully utilizing the original test image, we help the model make better guesses to learn from, which ultimately leads to more accurate final decisions. Our findings reveal what AI models actually learn when facing unfamiliar images, and show that prioritizing the original image creates much better learning targets. To help other researchers explore this idea, we have released a combination strategy called SE and a complete framework called USE.