Autoregressive Image Generation with Masked Bit Modeling

This paper challenges the dominance of continuous pipelines in visual generation. We systematically investigate the performance gap between discrete and continuous methods. Contrary to the belief that discrete tokenizers are intrinsically inferior, we demonstrate that the disparity arises primarily from the total number of bits allocated in the latent space (i.e., the compression ratio). We show that scaling up the codebook size effectively bridges this gap, allowing discrete tokenizers to match or surpass their continuous counterparts. However, existing discrete generation methods struggle to capitalize on this insight, suffering from performance degradation or prohibitive training costs with scaled codebook. To address this, we propose masked Bit AutoRegressive modeling (BAR), a scalable framework that supports arbitrary codebook sizes. By equipping an autoregressive transformer with a masked bit modeling head, BAR predicts discrete tokens through progressively generating their constituent bits. BAR achieves a new state-of-the-art gFID of 0.99 on ImageNet-256, outperforming leading methods across both continuous and discrete paradigms, while significantly reducing sampling costs and converging faster than prior continuous approaches. Project page is available at https://bar-gen.github.io

Modern AI image generators usually shrink images into a simpler internal form before learning how to create new ones. Many leading systems use continuous representations, while discrete representations—where an image is broken into codes more like words in a sentence—are often viewed as less accurate. This work shows that the gap is mostly not because discrete codes are inherently worse, but because they have typically been given too little capacity to store visual information. The paper compares discrete and continuous image representations using a shared measure: how many bits are available to describe the image. When discrete representations are given a similar bit budget, they can match or even outperform continuous ones. The challenge is that very large sets of discrete codes are hard for standard image generators to predict. To address this, the authors introduce BAR, a method that predicts each image code by gradually filling in its binary bits instead of choosing from an enormous list of possible codes all at once. This makes high-quality discrete image generation more scalable and efficient. BAR achieves state-of-the-art image quality on ImageNet-256 while sampling faster than many strong existing methods, suggesting that discrete image generation is a promising alternative to today’s dominant continuous pipelines.