Variational Adapter for Cross-modal Similarity Representation

The core of vision-language models lies in measuring cross-modal similarity within a unified representation space. However, most image-text matching or multi-class image classification datasets lack fine-grained cross-modal matching annotations, forcing the continuous similarity space into binary classification boundaries. This compression induces false negative samples and significantly impairs the generalization performance of cross-modal tasks. While prior research has attempted to mitigate this by modeling intra-modal ambiguity, it often overlooks inherent annotation flaws, leading to suboptimal uncertainty allocation. To address these challenges, we propose a Variational Adapter for Cross-modal Similarity Representation (VACSR). This approach reformulates image-text matching with fine-grained semantic scarcity as a variational inference problem. It constructs a latent space for cross-modal similarity and uses regularization techniques to mitigate overfitting to binary annotations. Experiments on image-text retrieval, domain generalization, and base-to-novel generalization demonstrate the proposed method’s effectiveness and robust generalization ability.

Vision-language models help computers connect images with text, such as finding the picture that best matches a caption or recognizing objects described by words. A key challenge is that many training datasets only tell the model whether an image and a text are a match or not. In reality, the relationship between an image and a sentence is often more gradual: some descriptions may be partly correct or closely related even if they are not labeled as exact matches. This paper studies how to better represent these gradual similarities between images and text. We propose a method called VACSR, which allows the model to learn a more flexible similarity representation instead of relying too heavily on simple yes-or-no labels. By doing so, the model can reduce the negative effects of imperfect annotations and avoid treating all unmatched examples as equally wrong. We evaluate our method on several image-text and image classification tasks. The results show that VACSR improves the model’s ability to retrieve relevant images or texts and helps it generalize better to new datasets, domains, and categories.