D$^2$O: A Dual Debiasing Operator for Training-Free Test-Time Adaptation of Vision–Language Models

Training-free test-time adaptation (TTA) for vision-language models (VLMs) can improve zero-shot classification under mild shifts, but often degrades under severe style/environment variation. We identify two shared failure modes in training-free pipelines: (i) retrieval confounding, where feature similarity is dominated by nuisance/style variation and corrupts retrieval evidence; and (ii) environment-biased priors, where VLM logits exhibit environment-dependent centered shifts that distort gating and prior-like terms. Therefore, we propose D$^2$O, a training-free debiasing operator that outputs three inference-time objects: a retrieval-oriented content feature for semantic matching, a style-aware routing coordinate for bias tracking, and debiased logits for corrected priors. D$^2$O composes plug-and-play with retrieval-based and closed-form Gaussian adapters in online and transductive settings. We further provide operator-to-decision guarantees: finite-difference covariance recovers a nuisance-sensitive subspace, routing-based EMA controls centered-logit bias estimates, and these errors yield bounded posterior log-odds perturbations, leading to a margin-based condition for label invariance. Extensive experiments show that D$^2$O achieves its clearest gains under style/environment-dominant shifts, with broader gains elsewhere. Code is available at \url{https://github.com/MAiTL-Group/D2O}.

Modern vision-language AI systems can recognize images by comparing them with text descriptions. They often work well on clean images, but their predictions can become unreliable when the test images look different from the data they were originally trained on, such as changes in lighting, background, texture, viewpoint, or image corruption. This paper studies how to improve such systems at test time without retraining the model, updating its parameters, or using the original training data. We find that two common problems can hurt existing training-free adaptation methods. First, the system may retrieve visually similar examples that match the background or style rather than the true object category. Second, the model’s prediction scores may be biased by the surrounding environment, making some wrong classes appear overly likely. To address these issues, we propose D$^2$O, a lightweight inference-time correction method. For each test image, D$^2$O produces three useful signals: one for more reliable visual matching, one for tracking the current visual environment, and one for correcting biased prediction scores. These signals can be added to existing training-free adaptation methods without changing their internal models. Experiments show that D$^2$O is most helpful when image style or environment changes are strong, while also providing smaller but consistent improvements on other shifted, corrupted, and fine-grained image recognition benchmarks.