Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching

Disaggregated serving alleviates memory bottlenecks in Large Language Model (LLM) inference but creates a severe communication bottleneck: transmitting high-dimensional Key-Value (KV) caches often dominates time-to-first-token (TTFT). Moreover, reusing caches across heterogeneous models (e.g., base and fine-tuned variants) causes semantic misalignment that accumulates over layers, degrading generation quality. We propose Semantic Cache Distillation (SCD), a loss-constrained framework that replaces raw KV transmission with compact semantic codes. SCD addresses these challenges via two mechanisms: (1) Reuse, which reconstructs most layers from low-rank subspaces to minimize transfer cost, and (2) Patch, which predicts normalized inputs at sparse transition layers to truncate error propagation. Empirically, SCD delivers up to 2.65 $\times$ TTFT speedup over the oracle consumer prefill and dominates quantization and selective recomputation baselines on the quality--latency Pareto frontier in bandwidth-constrained regimes, while keeping generation quality within 5\% F1 of the oracle.

When an AI assistant receives a long question or document, it first builds an internal working memory of what it has read. In large AI services, that reading may be done by one computer, while a different computer or a specialized version of the model writes the answer. A simple way to connect them is to send the whole working memory, but that can be so large that it delays the first word of the reply. Sending a rough shortcut is faster, but small mismatches between the two models can grow into wrong answers. We study how to pass this working memory more carefully between related models. Our method sends a compact version of the parts that are easy to share, then repairs a few sensitive places where errors are most likely to spread. This lets the second model begin writing sooner while keeping its answer close to what it would have produced if it had read everything itself. The result is a step toward faster and more efficient AI services, especially when many specialized models need to reuse the same long input.