Vector Linking via Cross-Model Local Isometric Consistency

We study Vector Linking: given two embedding clouds produced by different black-box encoders over partially overlapping datasets, recover cross-model object correspondences using only vectors. Empirically and theoretically, we show that independently trained contrastive encoders exhibit local geometric consistency: short-range distances are approximately preserved up to a scale factor, while long-range distances are not due to model-specific distortion. Building on this, we propose an iterative, reference-based geometric embedding hashing that recovers vector links from a tiny seed set of paired anchors. It represents each vector by distances to sampled paired anchors, proposes candidate links via hash-space matching, and aggregates evidence across views in a Beta--Bernoulli posterior to bootstrap high-confidence links as new anchors. Experiments across multiple benchmarks and embedding model pairs demonstrate accurate and robust linking under varying overlap, seed budgets, and out-of-domain references, with applications to vector database integration and cross-model clustering. Code is available at \url{https://github.com/DBgroup-Edinburgh/VecLinking}.

Many applications store documents or other items as embedding vectors, so that similar items are close together. A practical problem arises when two databases are built with different embedding models: the vectors live in different coordinate systems, so we cannot directly tell which vector in one database represents the same item as a vector in the other. This paper studies how to find these matches using only the stored vectors and a very small number of known matched examples. We find that, although two models may arrange a database differently, they often preserve small local neighborhoods around each item. Our method uses the known matches as anchors, compares how far other vectors are from small groups of anchors, combines many local votes, and gradually adds only high-confidence matches. Experiments on several text benchmarks show that this approach can recover many correct links and can help merge or search vector databases built with different embedding models.