Scalable GANs with Transformers

Scalability has driven recent advances in generative modeling, yet it remains underexplored for adversarial learning. We study the scaling behavior of Generative Adversarial Networks through two design choices: training in a compact Variational Autoencoder latent space and using purely transformer-based generators and discriminators. While this setup is efficient and scales well with compute, naively scaling exposes failure modes; underutilization of early layers in the generator and increasing optimization instability. We address these issues with lightweight intermediate supervision and width-aware learning-rate adjustment. Our Generative Adversarial Transformers (GAT) train reliably from small (S) to extra-large (XL) model sizes, and GAT-XL model achieves state-of-the-art single-step class-conditional generation on ImageNet at 256×256 resolution (FID of 2.18) in 60 epochs, requiring 4x fewer epochs than strong baselines.

AI image generators are becoming more powerful, but many of today’s strongest systems require multiple refinement steps to create a single image. This can make image generation slower and more expensive, especially when models are used at large scale or in interactive tools. GANs offer an appealing alternative because they can generate an image in one quick pass, but they have not benefited from scaling as reliably as newer generative models. Our work asks whether GANs can be made scalable again. We show that, with the right design choices, GANs can grow from small to very large models while continuing to improve. A key challenge is that larger GANs do not automatically use all parts of the model effectively and can become unstable during training. We address this by helping the model build images progressively, from rough structure to fine detail, and by keeping training behavior consistent as the model becomes wider. As a result, our largest model produces high-quality images in a single step and reaches strong performance with fewer training epochs than competing one-step methods. These results suggest that GANs are still a viable path for fast, efficient, and editable image generation. More broadly, our work reopens the question of how far adversarial learning can scale in modern generative AI.