Scaling Transformers for End-to-End Discrete Audio Tokenization

Discrete audio tokenizers are fundamental to empowering large language models with native audio processing and generation capabilities. Despite recent progress, existing approaches often rely on pretrained encoders, semantic distillation, or heterogeneous CNN-based architectures. These designs introduce fixed inductive biases that limit reconstruction fidelity and hinder effective scaling. In this paper, we argue that discrete audio tokenization should be learned fully end-to-end using a homogeneous and scalable architecture. Based on this perspective, we propose $\textbf{TAC}$, a Transformer-based audio tokenizer that jointly optimizes the encoder, quantizer, and decoder from scratch for high-fidelity reconstruction of general audio. We show that a simple, fully end-to-end learned tokenizer built from homogeneous, causal Transformer blocks scales gracefully and supports high-fidelity reconstruction across diverse audio domains. Across speech, sound, and music, the proposed tokenizer consistently outperforms prior codecs over a wide range of bitrates, while exhibiting predictable improvements with increased scale. Notably, leveraging TAC’s discrete tokens, we develop the first purely autoregressive TTS model that surpasses prior non-autoregressive and cascaded systems. Furthermore, TAC enables competitive ASR performance without auxiliary encoders. Our findings position TAC as a unified, scalable interface for the next generation of native audio foundation models.

Modern AI systems are very good at handling text, but making them understand and produce audio is harder because sound contains fine details such as pitch, speaker identity, background noise, and music. A common approach is to first turn audio into a shorter sequence of symbols that an AI system can work with, but existing methods often depend on separate pretrained parts or designs that do not improve easily as more data and computing power are used. We built TAC, a simpler audio representation system trained as one whole system from raw audio, so it learns how to preserve both meaning and sound quality. TAC can represent speech, everyday sounds, and music at different compression levels while still reconstructing audio with high quality. We found that increasing the system size and the amount of audio data consistently improves results. The same audio representation also helps build stronger systems for turning text into speech and for transcribing speech into text. This suggests that one shared way of representing audio could support compression, speech generation, and audio understanding in future AI systems. Because better speech generation can also be misused for impersonation, careful release practices and consent-aware use are important.