CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering

Emotional expression in human speech is nuanced and compositional, often involving multiple, sometimes conflicting, affective cues that may diverge from linguistic content. In contrast, most expressive text-to-speech (TTS) systems enforce a single utterance-level emotion, collapsing affective diversity and suppressing mixed or text–emotion–misaligned expression. While activation steering via latent direction vectors offers a promising solution, it remains unclear whether emotion representations are linearly steerable in TTS, where steering should be applied within hybrid TTS architectures, and how such complex emotion behaviors should be evaluated. This paper presents the first systematic analysis of activation steering for emotional control in hybrid TTS models, introducing a quantitative, controllable steering framework, and multi-rater evaluation protocols that enable composable mixed-emotion synthesis and reliable text–emotion mismatch synthesis. Our results demonstrate, for the first time, that emotional prosody and expressive variability are primarily synthesized by the TTS language module instead of the flow-matching module, and also provide a lightweight steering approach for generating natural, human-like emotional speech.

Human emotions are complex and often mixed. A person might feel relief and sadness simultaneously, or say "I'm fine" while their voice reveals frustration. Current text-to-speech systems still treat emotion as a single label per sentence, producing speech that lacks the richness of real human expression. We developed CoCoEmo, a method that enables composable and controllable emotion in existing speech synthesis models without any retraining. By identifying where emotional information is encoded inside these models and injecting directional signals, users can specify precise emotion blends (e.g., 60% happy, 40% sad), adjust emotional intensity, or make the voice convey an emotion that differs from what the words suggest. This plug-and-play approach could enable more expressive voice assistants, richer audiobook narration, and better assistive technologies, by allowing fine-grained control over how synthetic voices express emotion, bringing them closer to the nuance of real human speech.