Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage

End-to-end speech-in, speech-out dialogue systems are emerging as a powerful alternative to traditional ASR–LLM–TTS pipelines but remain prone to hallucinations due to limited factual grounding. While text-based dialogue models have effectively mitigated this issue through tools such as web search APIs, extending such capabilities to speech-in, speech-out systems remains underexplored. A key challenge is that tool integration increases latency, disrupting conversational flow. To mitigate this, we propose Streaming Retrieval-Augmented Generation (Stream RAG), a novel framework that reduces latency by predicting tool queries in parallel with user speech, even before the user finishes speaking. Specifically, we develop a post-training pipeline that teaches the model when to issue tool calls and how to generate spoken summaries using retrieved text results, thereby improving both accuracy and responsiveness. To evaluate our approach, we construct AudioCRAG, a benchmark created by converting queries from the publicly available CRAG dataset into speech form. Experimental results show that Stream RAG improves QA accuracy by over 20.0% absolute on AudioCRAG and achieves state-of-the-art performance, including outperforming cascaded systems, on the SLUE-SQA benchmark, while reducing latency by up to 57%. Stream RAG is modality-agnostic and can be applied equally to typed input, paving the way for more agentic, real-time AI assistants.

Voice assistants are becoming increasingly capable, but they still face a common problem: when asked questions that require up-to-date information, they often need to look up information online before responding. This can create noticeable delays that interrupt the natural flow of conversation. As a result, users are often forced to choose between fast responses and accurate ones. In this work, we introduce a new approach that allows a voice assistant to begin gathering information while a person is still speaking, rather than waiting until the entire question has been finished. By overlapping information gathering with listening, the assistant can respond more quickly without sacrificing accuracy. To evaluate this idea, we created AudioCRAG, a new benchmark of spoken questions that require external information to answer. The benchmark includes both synthetic and human-recorded speech and is designed to test whether voice assistants can remain both accurate and responsive when answering knowledge-seeking questions. Our experiments show that the proposed approach improves answer quality while significantly reducing response delays compared to existing systems. In human evaluations, people consistently preferred our system because it felt faster and provided more accurate answers. We believe these advances can help enable a new generation of voice assistants for phones, smart glasses, wearable devices, and customer-service applications that can listen, search for information, and respond naturally in real time.