Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models

Foundation models have driven rapid progress in computer vision, yet the two dominant paradigms, vision-language foundation models (VLMs) and vision-only foundation models (VFMs), remain only partially compatible. VLMs offer language-grounded semantic alignment but are often visually coarse, while VFMs learn discriminative perceptual geometry but lack semantic grounding. We propose GPUA, a Geometry-Preserving Unsupervised Alignment framework that integrates the complementary strengths of VFMs and VLMs. Inspired by cross-lingual alignment, GPUA treats VFM features as a visual language and learns an orthogonal mapping that translates the VFM space into the VLM semantic space, preserving geometry and narrowing the modality gap without labels or model parameter updates. GPUA is task-agnostic and requires only feature-level access to pretrained models. Experiments across diverse benchmarks demonstrate improved cross-model compatibility and strong gains in downstream zero-shot recognition and segmentation with negligible overhead. Our code is available at: https://github.com/Yuteam14/GPUA.

Modern computer vision mainly relies on two types of foundation models. Vision-language models can connect images with human language, but they often overlook fine-grained visual details. In contrast, vision-only models are effective at capturing detailed visual structures, yet they lack semantic understanding. These two types of models have complementary strengths, and effectively combining them could enable both fine visual perception and high-level semantic understanding. To address this problem, we propose GPUA, an unsupervised alignment framework that does not require labeled data. Our key idea is to treat the features learned by vision-only models as a kind of “visual language” and translate them into the semantic space of vision-language models through a geometry-preserving mapping. This allows different models to share information while retaining important visual structures. Experiments show that GPUA improves compatibility between different foundation models and leads to better performance on tasks such as zero-shot recognition and open-vocabulary segmentation, with almost no additional computational cost. Our work suggests that different AI models, much like people speaking different languages, can collaborate more effectively through translation-based alignment.