AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing

Despite recent breakthroughs, audio foundation models struggle in processing complex multi-source acoustic scenes. We refer to this challenging domain as audio stories, which can have multiple speakers and background/foreground sound effects. Compared to traditional audio processing tasks, audio stories introduce new layers of semantic, temporal, and physical complexity. To address this challenge, we propose AudioChat, a framework for developing audio foundation models that can generate, edit, and understand audio stories. AudioChat introduces a new paradigm in which LLM-based toolcalling agents simulate interactions between users and the system, and these simulated dialogues are used as training data. We also introduce a novel Audio Transfusion Forcing objective to train the AudioChat model, allowing it to simultaneously decompose high-level instructions via structured chain-of-thought reasoning and perform interactive multi-turn audio understanding/generation. To evaluate generation and editing performance, we develop three new metrics that directly measure task performance instead of relying upon distribution-based scoring. We highly encourage readers to visit our demo to better understand the capabilities of AudioChat: https://audiochat-icml-2026.github.io/.

Audio editing is the task of manipulating the content of an audio given an instruction (remove the dog, add a cat, make the man quieter, etc). Existing models for this task primarily rely on randomly generated audio mixtures, they take random audio clips and splice them together. This leads to two problems: 1.) unrealistic audio scenes and 2.) the inability to train on complex editing instructions. We propose a new method to generate synthetic audio editing training data using modern generative models, and a new technique to train models that can perform complex editing via reasoning.