KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models

Vision-language foundation models such as CLIP and SigLIP provide widely used representations for multimodal learning systems. While these models are typically compared through downstream performance, such evaluations often do not explain how their representations differ structurally. In this work, we study this problem through the task of Contrastive Embedding Clustering: identifying sample subsets that are weakly clustered under one representation but strongly clustered under another. We propose Kernel Optimization for Discrepancy Analysis (KODA), a kernel-based framework for contrastive representation comparison and alignment. KODA constructs unified multimodal kernels through modality-wise kernel composition and formulates discrepancy discovery as a constrained optimization problem that searches for coherent structures in one representation while suppressing coherence in a reference representation. This yields interpretable discrepancy directions associated with specific sample subsets and modality interactions. To scale KODA to large vision-language datasets, we develop randomized low-dimensional approximations of joint kernels using random projections, including Random Fourier Features for shift-invariant kernels. Empirically, KODA identifies consistent and interpretable discrepancy structures across vision-language representations and provides sample subsets for representation alignment. The code is available at https://github.com/yokiwuuu/KODA.

Vision-language models such as CLIP and SigLIP help AI systems connect images and text, and they are now widely used in tasks such as image search, classification, and multimodal understanding. These models are usually compared by benchmark scores, but such scores do not explain how two models organize the same images and captions internally. In this work, we study how to make this comparison more explainable. We ask whether we can find concrete groups of samples where two models behave differently: for example, samples that are clearly grouped by one model but mixed together by another. We introduce KODA, a method for explainable representation comparison that searches for these contrasting sample groups by comparing the similarity patterns produced by two models. KODA can be applied to both image-only and image-text data, and it is designed to scale to large datasets. In our experiments, it finds meaningful and interpretable data slices where vision-language models differ in their internal organization. These slices can help researchers better understand model behavior, diagnose model-specific weaknesses, and select samples for targeted representation alignment. Our code is publicly available at https://github.com/yokiwuuu/KODA.