SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport

The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world. Recent work exploits this convergence by aligning frozen pretrained vision and language models with lightweight alignment layers, but typically relies on contrastive losses and millions of paired samples. In this work, we ask whether meaningful alignment can be achieved with substantially less supervision. We introduce a semi-supervised setting in which pretrained unimodal encoders are aligned using a small number of image-text pairs together with large amounts of unpaired data. To address this challenge, we propose SOTAlign, a two-stage framework that first recovers a coarse shared geometry from limited paired data using a linear teacher, and then refines the alignment on unpaired samples via an optimal-transport-based divergence that transfers relational structure without overconstraining the target space. Unlike existing semi-supervised methods, SOTAlign effectively leverages unpaired images and text, learning robust joint embeddings across datasets and encoder pairs, and significantly outperforming supervised and semi-supervised baselines.

Artificial intelligence models are often trained separately on different kinds of data such as images or text. Recent work suggests that these models may independently learn similar views of the same underlying world. For example, a vision model may learn that dogs and cars are visually related in ways that partly match how a language model describes them. In this work, we ask whether this natural similarity of models can be leveraged to connect them without requiring millions of image-text pairs for training. This connection is important for tasks such as retrieval, where a system should find the right image for a given description, or the right description for a given image. We introduce SOTAlign, a method that aligns pretrained image and text models using only a small number of paired examples, together with many unpaired images and texts. SOTAlign first uses the paired data to build a rough bridge between the two models and then improves this bridge by comparing broader patterns in unpaired image and text collections. Our results show that unpaired data can substantially improve vision-language alignment, making it easier to reuse powerful pretrained models when paired data is scarce.