Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models

Spoken Language Models (SLMs) have emerged as a promising paradigm for speech synthesis by bypassing explicit grapheme-to-phoneme pipelines. However, their effectiveness in low-resource languages remains fundamentally limited by the scarcity of transcribed speech. In practice, synthetic data has become the primary strategy for scaling SLMs in such settings, providing reliable phonetic supervision when real data is insufficient. In this work, we show that this reliance introduces a fundamental trade-off, which we term the Stability-Expressivity Gap: while synthetic data improves phonetic accuracy, it progressively suppresses prosodic variability, ultimately leading to a collapse of expressivity (Synthetic Erosion). To bridge this gap, we propose two self-alignment frameworks. Disentanglement-Guided Self-Alignment (DGSA) recovers expressivity for complex languages by exploiting prosody-timbre separation. For regimes where authentic references are exceptionally limited, Temperature-Driven Self-Critique (TDSC) stabilizes generation through automated exploration and filtering. Our approach outperforms strong commercial systems, including ElevenLabs and Gemini Pro, and enables the first zero-shot voice cloning capability for Lao. Audio Samples are available at: https://luoji.cn/static/multilantts-demo-main/.

Many languages still lack high-quality speech synthesis systems because there is not enough recorded and transcribed speech to train them. A common solution is to create artificial training speech, but our study shows that this approach has a hidden trade-off: a moderate amount of synthetic speech helps the system pronounce words more accurately, while too much synthetic speech makes the generated voice sound flatter, less natural, and more repetitive. To address this problem, we develop two training methods that help speech generation systems keep both accuracy and naturalness. For Thai, our method uses real speech examples to guide the model toward more expressive and stable speech. For Lao, where real speech data is especially limited, our method lets the model generate multiple attempts, automatically filter out poor results, and improve itself over several rounds. Experiments show that our approach improves speech quality for both Thai and Lao and performs better than several strong existing systems. More broadly, this work provides a practical path toward better speech technology for languages that are currently underserved by modern AI systems.