Expressive Hindi Audiobook Generation with CLAP-Based Retrieval

Open-weight text-to-speech models for Hindi produce fluent narration but fail to adapt prosody to narrative context, generating every sentence in a near-identical tone regardless of emotional or dramatic content. We address this with a retrieval-augmented pipeline that requires no fine-tuning of the synthesis model. We train a Hindi CLAP model (MuRIL text encoder, HTS-AT audio encoder) on IndicVoices and Rasa to learn a joint text-audio embedding space. At inference, each sentence is embedded and matched against a precomputed library of Hindi audio clips via cosine similarity. The retrieved audio conditions IndicF5, a zero-shot TTS model that accepts reference waveforms natively, for expressive speech generation. Sentence-level outputs are then concatenated into full audiobook narrations. Our pipeline achieves a MOS of 4.00, NMOS of 3.89, and state-of-the-art intelligibility with 17.07% WER and 5.19% CER on held-out Hindi test data. The full pipeline is open-weight and runs on a single GPU.