PADS-TAL: Padding-Annealed Diffusion Sampling in Text-Aware Latent Space for Robust and Diverse Text-to-Music Generation

Text-to-Music diffusion models are increasingly used in real-world applications, yet deployment remains challenging: generations can collapse to limited patterns even with diverse initial noise and prompts, and inference-time diversity control often harms text alignment and fidelity by distorting key prompt cues established in early denoising. To address this, we propose Padding-Annealed Diffusion Sampling, which perturbs only a padding-indexed subspace while keeping non-padding conditioning fixed, enabling controlled exploration with reduced semantic drift. However, in a text-unaware VAE latent space, such exploration is less likely to stay within genre-faithful neighborhoods, limiting genre-consistent diversity. We therefore introduce Text-Aware Latent space that aligns local neighborhoods with text-implied genre structure, promoting genre-consistent diversity. Together, the two techniques form a unified pipeline that, compared to prior methods that perturb the full conditioning, achieves a better text alignment--diversity trade-off: at comparable text alignment, it delivers 15.4\% higher diversity with a relatively small fidelity drop, and further improves within-genre diversity by 71.6\%. Generated samples are available at https://pads-tal.github.io/PADS-TAL.io

Text-to-music models are becoming useful for creative applications, but they still struggle to generate multiple diverse pieces that all faithfully follow the same prompt. Even when users provide different prompts or random seeds, the outputs can collapse to similar musical patterns; when stronger diversity controls are used, the music may drift away from key prompt cues such as genre, mood, or instruments. This limits their usefulness in real-world settings where users often want several different musical options that still match their request. We propose a two-part method to improve this balance. First, our sampling method adds controlled randomness only to parts of the text representation that do not correspond to meaningful prompt words, reducing the chance of changing the intended meaning. Second, we improve the model's internal music representation so that different generations are more likely to stay within the genre or style implied by the prompt. Together, these techniques help the model produce more diverse music while better preserving the user's prompt. In our experiments, the method improves diversity at comparable text alignment compared with prior approaches, with generated examples available at https://pads-tal.github.io/PADS-TAL.io.