The Convergent Representation of Contrastive Vision-Language Models: Geometry, Modality Gap and Shared Space Alignment

Multimodal contrastive learning (MCL) aims to embed data from two modalities in a shared embedding space. However, in practice, image and text representations occupy completely separated regions of the embedding space, a phenomenon called the modality gap. Meanwhile, empirical findings on how the modality gap affects downstream performance remain inconsistent. These observations motivate two key questions: (1) What causes the modality gap? (2) What determines downstream performance? To address these questions, we develop the first theoretical framework for analyzing the geometry of convergent optimal representations (COR) of MCL when training is optimized. We prove that the modality gap emerges when image and text representations collapse into different subspaces, a phenomenon called \emph{dimension collapse}. Our theory further reveals that although the modality gap prevents direct alignment between image and text representations, their projections onto the shared subspace can align. Moreover, we show that shared space alignment is a dominant factor in downstream performance, while the effect of the modality gap is limited. Inspired by these findings, we propose Shared Space Alignment (SSA) to improve MCL pretraining by enhancing alignment in the shared space without optimizing for modality gap reduction. Extensive empirical results validate our theoretical analysis and the proposed method.

Vision-language models learn by matching images with their corresponding text descriptions in a shared representation space. However, in practice, image and text representations often remain separated from each other, a phenomenon known as the modality gap. It has been unclear why this gap appears and whether reducing it is actually important for downstream performance. This paper studies these questions from a geometric perspective. We show that the modality gap is not mainly caused by simple cone-shaped concentration of the two modalities, but instead arises when image and text representations collapse into different, partially overlapping subspaces. Our analysis further shows that even when image and text representations are separated in the original space, their projections can still align in a shared subspace. Across 78 pretrained vision-language models, we find that this shared space alignment is more strongly related to downstream performance than the size of the modality gap itself. Based on this insight, we propose Shared Space Alignment, a training objective that improves vision-language pretraining by directly strengthening alignment in the shared subspace rather than explicitly reducing the modality gap.