LEMUR: Learned Multi-Vector Retrieval

Multi-vector representations generated by late interaction models, such as ColBERT, enable superior retrieval quality compared to single-vector representations in information retrieval applications. In multi-vector retrieval systems, both queries and documents are encoded using one embedding per token, and similarity between queries and documents is measured by the MaxSim similarity measure. However, the improved quality of multi-vector retrieval comes at the expense of significantly increased search latency. In this work, we introduce LEMUR, a simple yet efficient framework for multi-vector similarity search. LEMUR consists of two consecutive problem reductions: First, we formulate multi-vector similarity search as a supervised learning problem that can be solved using a one-hidden-layer neural network. Second, we reduce inference under this model to single-vector similarity search in its latent space, enabling the use of existing single-vector search indexes to accelerate retrieval. LEMUR is an order of magnitude faster than prior multi-vector similarity search methods. Our code is available at https://github.com/ejaasaari/lemur

Many search systems use a simple shortcut: they compress each document into one compact representation that can be compared with a compact representation of a user query. This makes searching very fast, but it can lose fine details. More fine-grained search systems compare query and document representations piece by piece, capturing subtle matches, but they are usually too slow for large collections. Our paper introduces LEMUR, a method that makes this fine-grained search much faster. LEMUR also compresses each document into one compact representation, but the key difference is how that representation is learned. Rather than acting as a generic summary, it is trained to behave like the more careful piece-by-piece comparison would behave when judging that document. LEMUR uses these learned compact representations to quickly find the most promising documents with existing fast search systems, then applies the full fine-grained comparison to only those documents. This makes it easier to build search systems that are both accurate and fast. We have released LEMUR as a free and easy-to-use system that anyone can build on.