CrossQ: Task-Aligned Cross-Token Conditional Quantization for Late Interaction Retrieval

Late-interaction retrievers like ColBERT achieve high quality but suffer from large multi-vector indices. Standard compression minimizes token reconstruction error, while ranking depends critically on preserving scores of sparse "winner" tokens. We introduce CrossQ, which adaptively improves effective token fidelity within documents by conditioning token codes on lightweight document context computed at indexing time (but not stored). CrossQ is trained with ranking-aligned objectives that preserve candidate score distributions and protect hard-negative margins. At 2 B/token, CrossQ improves MRR@10 by +0.010 over the strongest strictly footprint-matched quantization baseline and by +0.012 over the strongest candidate-matched system reference. On the nine-dataset BEIR subset reported in Appendix G.1, CrossQ improves average nDCG@10 by +0.009 at 4 B/token over the strongest candidate-matched system reference. At 4 B/token, CrossQ achieves $64\times$ raw token-storage reduction, approximately $61\times$ including metadata and approximately $58\times$ under conservative padding/alignment accounting. At 8 B/token, CrossQ + light fine-tuning retains approximately 98% of full-precision ColBERT MRR@10, improving the footprint-quality tradeoff for memory-constrained late-interaction retrieval.

Modern search systems can find relevant documents very accurately, but some of the strongest methods require storing a large amount of information for every document. This makes them expensive to run at large scale, especially when the search index must fit in memory for fast responses. This paper introduces CrossQ, a method for making these search indexes much smaller while keeping most of the search quality. Instead of compressing every stored document token independently, CrossQ looks at the document as a whole when deciding how to store each token. This helps preserve the parts of the document that are most likely to matter for ranking search results. CrossQ is trained to preserve the final search ranking, not just to reconstruct stored vectors accurately. In experiments on standard search benchmarks, CrossQ improves over strong compression baselines and keeps retrieval quality close to the full-size system. At one operating point, it reduces storage by about 61 times when metadata is included, or about 58 times under conservative padding and alignment accounting. These results suggest that high-quality neural search can be made more practical for memory-constrained deployment.