End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer

Autoregressive image modeling relies on visual tokenizers to compress images into compact latent representations. We design an end-to-end training pipeline that jointly optimizes reconstruction and generation, enabling direct supervision from generation results to the tokenizer. This contrasts with prior two-stage approaches that train tokenizers and generative models separately. We further investigate leveraging vision foundation models to improve 1D tokenizers for autoregressive modeling. Our autoregressive generative model achieves strong empirical results, including a state-of-the-art FID score of 1.48 without guidance on ImageNet 256×256 generation.

Modern AI image generators work much like language models: they build a picture piece by piece, predicting the next "visual token" in a sequence. They depend on a separate tool, a tokenizer, that first chops images into these pieces. The standard recipe trains the tokenizer alone to faithfully copy images, then freezes it and trains the generator on top. The tokenizer therefore ends up good at reconstruction, not at generation. We train the tokenizer and the generator together, end-to-end, so the generator can shape what the tokens mean. The catch is that the generator's learning signal lives in a discrete token space that doesn't directly reflect final picture quality; we bridge this with a new loss that decodes the generator's predictions back to pixels and grades them there. We also let the tokenizer borrow knowledge from a pretrained vision model so each token carries richer meaning. The result, EOSTok, sets a new state of the art on ImageNet image generation and scales gracefully, producing sharper, more coherent images with far fewer parameters than competing systems.