Polyphonia: Zero-Shot Timbre Transfer in Polyphonic Music with Acoustic-Informed Attention Calibration

The advancement of diffusion-based text-to-music generation has opened new avenues for zero-shot music editing. However, existing methods fail to achieve stem-specific timbre transfer, which requires altering specific stems while strictly preserving the background accompaniment. This limitation severely hinders practical application, since real-world production necessitates precise manipulation of components within dense mixtures. Our key finding is that, while vanilla cross-attention captures semantic features of stems, it lacks the spectral resolution to strictly localize targets in dense mixtures, leading to boundary leakage. To resolve this dilemma, we propose Polyphonia, a zero-shot editing framework with Acoustic-Informed Attention Calibration. Rather than relying solely on diffuse semantic attention, Polyphonia leverages a probabilistic acoustic prior to establish coarse boundaries, enabling non-target stems preserved precise semantic synthesis. For evaluation, we propose PolyEvalPrompts, a standardized prompt set with 1,170 timbre transfer tasks in polyphonic music. Specifically, Polyphonia achieves an increase of 15.5% in target alignment compared to baselines, while maintaining competitive music fidelity and non-target integrity.

Have you ever wanted to change a specific instrument in a fully mixed song, like turning the vocals into a violin, while keeping the background music completely untouched? For artificial intelligence, this is very difficult. Because different sounds overlap in a song, the AI often gets confused and accidentally alters the background accompaniment. To solve this, we created a system called Polyphonia. Instead of relying solely on the AI's understanding of text descriptions, we first use an audio separation tool to map out the exact physical boundaries of the target instrument. We then restrict the AI to only generate new sounds within these specific boundaries. This ensures that the target instrument is accurately transformed while the original rhythm and other background instruments are strictly preserved. Our approach works significantly better than existing methods in editing specific sounds within complex mixtures. To help other researchers evaluate this task, we also created a standardized set of over 1,100 text prompts (editing challenges) to be used with existing public music collections.