Simultaneous Speech-to-Speech Translation Without Aligned Data

Simultaneous speech translation requires translating source speech into a target language in real-time while handling non-monotonic word dependencies. Traditional approaches rely on supervised training with word-level aligned data, which is difficult to collect at scale and thus depends on synthetic alignments using language-specific heuristics that are suboptimal. We propose Hibiki-Zero, which eliminates the need for word-level alignments entirely. This fundamentally simplifies the training pipeline and enables seamless scaling to diverse languages with varying grammatical structures, removing the bottleneck of designing language-specific alignment heuristics. We first train on sentence-level aligned data to learn speech translation at high latency, then apply a novel reinforcement learning strategy using GRPO to optimize latency while preserving translation quality. Hibiki-Zero achieves state-of-the-art performance in translation accuracy, latency, voice transfer, and naturalness across four X-to-English tasks. Moreover, we demonstrate that our model can be adapted to support a new input language with less than 1000h of speech. We provide examples, model weights, inference code and we release a benchmark containing 45h of multilingual data for speech translation evaluation.

Simultaneous speech translation makes conversations across languages feel natural and immediate. However, building such systems is difficult because grammatical structures can differ significantly between languages: a real-time translator often needs to wait for crucial words before producing a correct translation. Existing systems therefore rely on word-level aligned training data to ensure temporal consistency between the source and translated speech. Creating such aligned data is costly and often depends on language-specific rules. We introduce Hibiki-Zero, a new approach for building simultaneous speech-to-speech translation systems that removes the need for fine-grained aligned data. Instead of training on word-level alignments, our system first learns from translations aligned at the sentence-level, which are much easier to obtain across many languages. We then use reinforcement learning to teach the model how to achieve the optimal trade-off between quality and latency: producing accurate translations while minimizing delay. Hibiki-Zero can process speech in French, Spanish, Portuguese, or German and simultaneously generate expressive translated speech in English. It also transfers characteristics of the original speaker’s voice and prosody to the translation, making the interaction more natural and engaging. Experimental results show improvements over previous state-of-the-art systems in translation quality, latency, speech naturalness, and speaker identity preservation. We further demonstrate that the approach can be adapted to a new language using fewer than 1000 hours of speech data, highlighting its scalability to additional languages, including lower-resource settings. By simplifying the training pipeline and removing the dependence on specialized data, our work makes high-quality real-time speech translation more accessible. We release our model, inference code, and a multilingual evaluation benchmark to support future research.