Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment

Neural speech codecs based on Vector-Quantized VAEs (VQ-VAEs) are core audio tokenizers for speech LLMs, yet their reconstruction fidelity is bottlenecked by quantization error. Modifying the quantizer or increasing model capacity are common fixes, but they complicate downstream language modeling. Our core idea is to align the decoder's internal feature manifolds when processing both the quantized tokens and their original continuous embeddings, using a lightweight feature-mapping loss. This requires minimal training overhead and no inference-time changes. Applied to XCodec2, self-guidance improves all reconstruction metrics, achieving state-of-the-art low-bitrate performance. Notably, it enables a 4× codebook reduction without fidelity loss, which downstream TTS experiments show significantly improves LLM-based synthesis by simplifying the token modeling space. Multiple statistical observations and visualizations corroborate the enhanced internal manifold alignment in the decoder. Extensive experiments confirm its generality across various inductive biases. Self-guidance thus establishes an efficient, broadly applicable method for high-fidelity neural audio coding.

Today’s AI voice assistants and translators compress speech into compact digital codes for fast transmission, then reconstruct it on the other side. The challenge is that compression is “lossy”—it introduces errors that can distort the reconstructed voice. We developed a training method called self-guidance that makes the reconstructing AI (the decoder) much better at ignoring these compression errors. Think of it like teaching a translator to understand someone with a heavy accent, without changing the speaker’s voice. During training, we have the decoder compare two versions of the same sound: the pristine original and the error-prone compressed version. By aligning its internal understanding of both, the decoder becomes far more robust—with no slowdown or changes when deployed. This approach works so well that we can shrink the “codebook”—the dictionary of sound fragments—to a quarter of its usual size without losing quality, enabling cheaper and faster transmission. We also found that this simpler code makes it much easier for large language models to generate natural-sounding speech. Since it is a simple training add-on, self-guidance could broadly improve future AI audio systems.