Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?

This paper investigates the scaling properties of autoregressive next-pixel prediction, a simple, end-to-end yet under-explored framework for unified vision models. Starting with images at resolutions of 32$\times$ 32, we train a family of Transformers using IsoFLOP profiles across compute budgets up to 7e19 FLOPs and evaluate three distinct target metrics: next-pixel prediction objective, ImageNet classification accuracy, and generation-based completion where top half of the image serves as a spatial prompt. First, optimal scaling strategy is critically task-dependent. At a fixed resolution of 32 $\times$ 32 alone, the optimal scaling properties for image classification and image generation diverge, where generation optimal setup requires the data size grow three to five times faster than for the classification optimal setup. Second, as image resolution increases, the optimal scaling strategy indicates that the model size must grow much faster than data size. Surprisingly, by projecting our findings, we discover that the primary bottleneck is compute rather than the amount of training data. As compute continues to grow four to five times annually, we forecast the feasibility of pixel-by-pixel modeling of images within the next five years.

Language models have achieved incredible success by simply predicting the next word in a sequence. While researchers have previously shown that vision models can use this exact same approach—predicting an image one raw pixel at a time —this method became less popular over the years in favor of more compute-efficient shortcuts. This paper asks a crucial unanswered question: what happens if we actually scale up this simple "next-pixel" approach? We conducted extensive experiments to uncover the scaling properties of learning from raw pixels, measuring how a model's performance improves as we add more data and computer power. We discovered three key insights: * Pixels are harder to scale than text: Because a single pixel holds far less meaning than a word, a model requires at least 10 to 20 times more data (tokens per parameter) at low resolution to successfully learn from raw pixels compared to language. * The end goal changes the scaling strategy: The optimal way to allocate resources depends heavily on the target task. Teaching a model to generate images requires the dataset size to grow three to five times faster than if you are teaching it to simply recognize or classify images. * Sharper images require much bigger models: As image resolution increases, simply feeding the model more pictures isn't enough; the model's size must grow much faster than the amount of training data to handle the added complexity. Ultimately, we found that the main roadblock to scaling this simple pixel-by-pixel approach is pure computing power, not a lack of training data. Because available computing power for frontier models continues to rapidly increase four to five times every year, we forecast that training highly capable vision models directly from raw pixels will become a feasible and practical reality within the next five years.