STAR-VAE: Structured Topology-Aware Regularization for Audio Reconstruction and Generation

Continuous Variational Autoencoders (VAEs) serve as the fundamental continuous tokenizer for modern neural audio generation systems, enabling high-fidelity reconstruction while providing a compact, smooth latent space for downstream generative priors. However, continuous VAEs face a fundamental conflict when balancing compression rate, reconstruction fidelity, and latent space topology—a challenge we formalize as the Rate-Distortion-Regularity Trilemma. This trilemma stems from a critical topological mismatch: the prevailing isotropic Gaussian prior in standard VAEs imposes a flat latent geometry that fails to accommodate audio's hierarchical nature, where low-frequency components are structured and compressible while high-frequency components are stochastic and incompressible, leading to disordered information packing where crucial semantic features are randomly interleaved with high-entropy noise. To resolve this challenge, we propose Structured Topology-Aware Regularization (STAR), a general training strategy that reshapes latent space geometry by imposing a growth-based constraint field, routing structural and textural information into channel subspaces with matching capacities. STAR is applicable to any VAE architecture and effectively resolves the trilemma, as demonstrated in CNN-based VAEs. To fully exploit STAR's potential, we present STAR-VAE, combining STAR with a hybrid CNN-Mamba architecture that synergizes local feature extraction with linear-complexity global context modeling, achieving state-of-the-art performance. We further propose STAR-Gen, an LLM-based Flow Matching framework that leverages STAR-VAE's structured latent space for high-fidelity generation without suffering from vector quantization artifacts. Empirical results demonstrate that STAR-VAE successfully resolves the trilemma, achieving state-of-the-art reconstruction fidelity and enhanced semantic information preservation across diverse audio domains. The structured latent space improves both traditional diffusion models and our STAR-Gen paradigm, achieving state-of-the-art performance in text-to-audio generation. The project page is available at https://STAR-VAE.github.io.

Modern AI systems that generate sound effects or music from text prompts rely on an "audio compressor" that squeezes raw sound into a compact internal form and back again — the final audio can only be as clean as this compressor allows. Today's compressors treat every part of a sound the same way, but sound is not uniform: a drumbeat or spoken word has clean, repeating structure, while the hiss of a wave or splash of water is essentially random texture. Forcing both into the same internal slots makes the compressor either blur the structure or hollow out the texture. Our method, STAR, is a simple training recipe that tells the compressor to reserve "premium" slots for structured, meaningful sounds and "budget" slots for noisy textures. The result is cleaner reconstructions and, when used by a downstream generator, noticeably more natural text-to-audio synthesis.