AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech

While existing text-to-speech (TTS) models exhibit high expressiveness, fine-grained control over composite instructions remains challenging due to the structural mismatch between discrete textual intents and continuous acoustic realizations. Inspired by human cognitive decoupling, we introduce AgentSteerTTS, a multi-agent closed-loop framework designed for intent-faithful expressive control of composite instructions. First, in our framework, an adversarial disentanglement module mitigates speaker-emotion leakage through gradient reversal and cross-covariance regularization. Next, a Dual-Stream Anchoring Controller grounds abstract intents using a large-scale acoustic prototype library: a Retrieval Agent selects expressive anchors, while a Synthesis Agent fuses them into continuous control vectors via gated attention. Finally, a Fast–Slow Feedback Agent refines output intensity through latent gradient correction and resolves semantic–acoustic mismatches using high-level perceptual critique. Experiments on a composite-instruction benchmark and public test sets show that AgentSteerTTS yields consistent and significant improvements to the baselines, demonstrating the effectiveness of the proposed method. Our code and models will be released.

Modern text-to-speech systems can generate natural speech, but they often struggle to follow complex style instructions that combine multiple emotions or speaking tones, such as sounding happy but slightly arrogant. AgentSteerTTS aims to make generated speech better match these mixed instructions while keeping the speaker’s voice recognizable and the audio natural. It separates speaker identity from expressive style, uses expressive speech examples to guide the desired style, and checks the generated result to adjust the expression when it is too weak, too strong, or incorrect. Experiments show that this approach follows complex speaking-style instructions more accurately than strong baseline systems while maintaining good speech quality.