MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation

To reduce LLM costs and latency, semantic caching systems must accurately identify when a new prompt matches a cached one. Current methods often rely on simplistic similarity measures, which limit their effectiveness. We introduce MVR-cache, a novel semantic caching approach that significantly improves retrieval accuracy by integrating Multi-Vector Retrieval (MVR). MVR-cache is built upon a learnable segmentation model that intelligently splits prompts, enabling fine-grained similarity comparisons via MaxSim. We derive the model's training objective from a rigorous theoretical analysis. This can ensure that optimizing this objective directly maximizes cache hits under strict correctness constraints. To solve the resulting non-differentiable combinatorial optimization problem, we leverage a reinforcement learning-based training strategy with the theoretically grounded objectives as the reward. Experimental results on established benchmarks across diverse tasks confirm that in comparison to the state-of-the-art, MVR-cache consistently increases the cache hit rates by up to 37\% while maintaining the same correctness guarantees. MVR-cache is available at https://github.com/PKU-SDS-lab/MVR-Cache

Large language model applications often receive prompts that differ in wording but require the same response, causing repeated model calls that increase cost and latency. Semantic caching reduces this redundancy by storing previous prompts and responses, then reusing a cached response when a new prompt is judged to be equivalent to a cached one. Existing semantic caching methods typically embed each entire prompt into a single vector and use cosine similarity between these embeddings to decide whether two prompts are similar enough. This single-vector comparison can miss fine-grained semantic differences, such as changes in sentiment, constraints, or requested output format, which may lead to either unnecessary model calls or unsafe response reuse. We introduce MVR-cache, a semantic caching method that represents each prompt using multiple vectors instead of one. MVR-cache learns how to split prompts into meaningful segments and compares these segments using multi-vector retrieval, enabling more precise matching between new and cached prompts. The method is trained with an objective designed to improve cache reuse while preserving strict correctness guarantees. Across benchmark tasks, MVR-cache improves cache hit rates by up to 37% over prior methods while maintaining the same correctness guarantees. By enabling more accurate and safe response reuse, MVR-cache can reduce the cost and latency of large language model applications without compromising reliability.