Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs

Vision-Language models (VLMs), such as CLIP, achieve powerful zero-shot classification. However, their predictions remain sensitive to spurious correlations, where contextual cues dominate over semantic content. Earlier solutions typically rely on fine-tuning or prompt engineering, which either undermine the advantages of pre-trained models or are prone to hallucination. In this work, we propose Density-Aware Translation (DAT) that refines image-text similarity scores using a local geometric density term derived from group reference sets. Our approach is motivated by the phenomenon that CLIP embeddings exhibit a modality gap and lie on an anisotropic shell in the feature space: common patterns cluster near the mean, while rare patterns are pushed outward. This geometry creates uneven alignment, where spurious correlations are amplified while semantically meaningful but rare cues are marginalised. To address this, we employ a relative measure to rescale similarities based on embedding density, suppressing overconfident scores in diffuse regions while preserving dense, semantically consistent matches. Experimental results on benchmark datasets demonstrate consistent improvements in worst-group and average accuracy, highlighting density-aware translation as a simple and effective calibration mechanism for reliable zero-shot classification using multimodal models.

Vision-language models such as CLIP can classify images using natural language descriptions, without being retrained for every new task. However, these models can sometimes rely on misleading patterns, such as background or context, instead of the object or concept that actually matters. For example, a model may strongly associate water with waterbirds and make mistakes when a waterbird appears on land. We propose Density-Aware Translation (DAT), a method that makes zero-shot vision-language models more robust without changing their parameters. DAT looks at how images are arranged in the model’s internal representation space and uses small reference sets to estimate which examples are typical or unusual for each group. It then adjusts image-text matching scores so that overconfident predictions driven by misleading correlations are reduced. Across several benchmark datasets and vision-language models, DAT improves performance on the worst-performing groups while maintaining strong average accuracy. This makes zero-shot vision-language models more reliable in settings where rare or underrepresented cases may otherwise be harmed by spurious correlations.