Predictive Prefetching for Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) improves factual grounding in large language models but suffers from substantial latency due to synchronous retrieval. While recent work explores asynchronous retrieval, existing approaches rely on heuristic coordination between retrieval and generation and assume stable information demands during decoding that often break in complex, multi-domain settings. In this paper, we propose an advanced asynchronous retrieval framework that enables predictive prefetching aligned with evolving information needs. The framework explicitly predicts when retrieval should be triggered and what information should be retrieved using three components, a retrieval predictor, a context monitor, and a query generator, by exploiting semantic precursors in generation dynamics that emerge several tokens before uncertainty becomes critical. Experiments on multiple benchmarks demonstrate up to 43.5% end-to-end latency reduction and 62.4% improvement in time-to-first-token, while maintaining answer quality comparable to synchronous RAG baselines.

Modern AI assistants often look up information from external sources like databases or the web to give accurate answers. This improves reliability but creates noticeable delays: each time the AI checks a fact, it pauses writing, waits for the lookup, then continues. Complex questions can require many such pauses. We asked whether an AI can anticipate what it will need and start fetching it in the background while it keeps writing, so users never see the delay. Our research shows this is possible. AI systems give off subtle signals in how they choose words and where they focus, roughly 8 to 16 words before they need outside information. We trained a small helper that spots these signals, decides what to search for, and fetches the answer in the background. Across four popular question-answering benchmarks, our approach delivered responses 43.5% faster overall and 62.4% faster to begin, with 31% fewer lookups and no loss in answer quality.