Unlocking Speech–Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning

Instruction tuning for speech language models (SLMs) is substantially more challenging than for text-based large language models (LLMs), as it requires learning a new modality and a wide range of speech-specific instructions in addition to those supported by text LLMs. Existing SLM training approaches largely replicate the text LLM training paradigm by synthesizing large-scale speech pre-training and instruction-tuning datasets. However, this strategy is difficult to scale, since speech sequences are significantly longer than text sequences. In this paper, we propose SpeechCombine, an instruction-following speech language model trained without any instruction tuning, using only a single round of speech pre-training on as little as 30k hours of speech data. Starting from a text LLM base model, we perform continuous pre-training on speech utterances to obtain a speech-adapted model, and then directly combine its weights with the weight difference between the instruction-tuned and base versions of the text LLM. Our results show that this simple combination strategy not only preserves the knowledge and capabilities of the original text LLM, but also effectively transfers them to the speech domain. These findings suggest a new direction for SLM training that avoids reliance on massive volumes of speech data.

Machine learning systems that understand speech usually require enormous amounts of spoken data and an additional training step called instruction tuning, where the model learns to follow human commands. This process is expensive and difficult because speech data is much larger and more complex than text. In this work, we introduce SpeechCombine, a new way to build speech-based AI systems without any speech instruction tuning. We start with a language model originally trained on text, adapt it to understand speech through a single round of speech training, and then transfer the instruction-following abilities of the text model into the speech model by combining their learned parameters. Despite its simplicity, our approach allows the model to follow spoken instructions while preserving much of the original knowledge and reasoning ability of the text model. Remarkably, this works using only about 30,000 hours of speech data — far less than many existing approaches require. Our findings suggest that powerful speech AI systems may be built more efficiently, reducing the need for massive speech datasets and making speech technology easier to develop and scale.