MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models

Speech-to-speech language models have recently emerged to enhance the naturalness of conversational AI. In particular, full-duplex models are distinguished by their real-time interactivity, including handling of pauses, interruptions, and backchannels. However, improving their factuality remains an open challenge. While scaling the model size could address this gap, it would make real-time inference prohibitively expensive. In this work, we propose Moshi-RAG, a modular approach that combines a compact full-duplex interface with selective retrieval to access more powerful knowledge sources. Our asynchronous framework enables the model to identify knowledge-demanding queries and ground its responses in external information. By leveraging the natural temporal gap between response onset and the delivery of core information, the retrieval process can be completed while maintaining a natural conversation flow. With this approach, Moshi-RAG achieves factuality comparable to the best publicly released non-duplex speech language models while preserving the interactivity inherent to full-duplex systems. Moreover, our flexible design supports plug-and-play retrieval methods without retraining and demonstrates strong performance on out-of-domain mathematical reasoning tasks.

Since its 2024 release, Moshi has pioneered natural, real-time AI conversation. However, like many fast-talking models, it can sometimes struggle with factual accuracy when faced with difficult questions. To address this, we developed MoshiRAG, which gives the model a "brain" for external research without slowing down the chat. Key features include: - Autonomous Retrieval: The model independently decides when it needs extra facts and fetches them in the background. - Seamless Integration: It incorporates new information into its speech without causing awkward pauses. - Full-Duplex Interactivity: It maintains the ability to listen and talk at the same time, reacting to you in real-time. - Plug-and-Play Design: The system can easily switch between different knowledge sources, like search engines or specialized databases, without being retrained. By combining the natural flow of Moshi with this flexible knowledge system, voice AI becomes significantly more accurate and trustworthy, even in complex areas like mathematical reasoning, while remaining fast and interactive. We have released our code and model weights to help the research community build voice assistants that are both conversational and reliably smart.