RAG without Forgetting: Continual Query-Infused Key Memory

Retrieval-augmented generation (RAG) systems commonly improve robustness via query-time adaptations such as query expansion and iterative retrieval. While effective, these approaches are inherently stateless: adaptations are recomputed for each query and discarded thereafter, precluding cumulative learning and repeatedly incurring inference-time cost. Index-side approaches like key expansion introduce persistence but rely on offline preprocessing or heuristic updates that are weakly aligned with downstream task utility, leading to semantic drift and noise accumulation. We propose Evolving Retrieval Memory (ERM), a training-free framework that transforms transient query-time gains into persistent retrieval improvements. ERM updates the retrieval index through correctness-gated feedback, selectively attributes atomic expansion signals to the document keys they benefit, and progressively evolves keys via stable, norm-bounded updates. We show that query and key expansion are theoretically equivalent under standard similarity functions and prove convergence of ERM’s selective updates, amortizing optimal query expansion into a stable index with zero inference-time overhead. Experiments on BEIR and BRIGHT across 13 domains demonstrate consistent gains in retrieval and generation, particularly on reasoning-intensive tasks, at native retrieval speed.

This paper introduces a new way for AI systems to improve how they retrieve information over time without needing expensive retraining. Modern AI assistants that use retrieval-augmented generation (RAG) often rely on temporary query rewriting or expansion to better search documents, but these improvements are discarded after each question. As a result, the system repeatedly performs the same costly retrieval adaptations and cannot accumulate experience. We propose Evolving Retrieval Memory (ERM), a lightweight framework that allows retrieval systems to “remember” successful search patterns by gradually updating document representations based on past successful queries. Instead of retraining models, ERM selectively stores useful query signals only when they improve retrieval or answer quality, enabling the retrieval index itself to evolve over time. This allows the system to continuously improve while maintaining the fast inference speed of standard retrieval systems. Across 13 benchmark domains covering scientific search, coding, mathematics, and reasoning-intensive tasks, ERM consistently improves both retrieval accuracy and downstream question-answering quality. These results suggest that retrieval systems can become more adaptive and efficient by learning from prior successful interactions without increasing inference-time cost.