EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments

Modern large language models (LLMs) extend context lengths to millions of tokens, enabling coherent, personalized responses grounded in long conversational history. However, the Key-Value (KV) cache grows linearly with the extended dialogue history, causing the model’s memory footprint to quickly exceed device limits. While recent KV cache compression methods attempt to reduce memory usage, most apply cache eviction after processing the entire context, incurring unbounded peak memory usage. Additionally, query-dependent eviction narrows the cache semantics to a single query, leading to failure cases in multi-turn conversations. In this paper, we introduce EpiCache, a training-free KV cache management framework for long conversational question answering (LongConvQA) under fixed memory budgets. EpiCache bounds cache growth through block-wise prefill and preserves topic-relevant context via episodic KV compression, which clusters conversation history into coherent episodes and performs episode-specific KV cache eviction. Across three LongConvQA benchmarks (LongMemEval, Realtalk, and LoCoMo), EpiCache improves accuracy by up to 30\%, achieves near-full-cache accuracy under $4$–$6\times$ compression, and reduces latency and peak memory by up to $2.4\times$ and $3.7\times$, respectively.

AI assistants on smartphones could one day remember weeks of past conversations to give truly personalized responses — but storing that history requires memory that grows without limit, quickly exceeding what a phone can handle. Existing approaches either ignore this memory constraint entirely or compress memory so aggressively that the assistant forgets critical details from earlier in the conversation. We introduce EpiCache, a framework that organizes long conversation history into topically coherent "episodes" — much like chapters in a book — and builds a compact, compressed memory snapshot for each episode. When a user asks a question, EpiCache identifies the most relevant episode and retrieves only that snapshot for generating a response, keeping total memory usage flat no matter how long the conversation grows. EpiCache further improves accuracy by intelligently distributing the limited memory budget across the AI model's layers, allocating more memory to layers that are most sensitive to compression. Across multiple real-world benchmarks, EpiCache improves answer accuracy by up to 30% over existing methods, reduces peak memory usage by nearly 4x, and cuts response time by 2.4x — marking a practical step toward capable, long-term conversational AI on resource-constrained devices.