Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation

Latent diffusion models excel at generating high-quality images but lose the benefits of end-to-end modeling. They discard information during image encoding, require a separately trained decoder, and model an auxiliary distribution to the raw data. In this paper, we propose Latent Forcing, a simple modification to existing architectures that achieves the efficiency of latent diffusion while operating on raw natural images. Our approach orders the denoising trajectory by jointly processing latents and pixels with separately tuned noise schedules. This allows the latents to act as a scratchpad for intermediate computation before high-frequency pixel features are generated. We find that the order of conditioning signals is critical, and we analyze this to explain differences between REPA distillation in the tokenizer and the diffusion model, as well as conditional and unconditional generation. Applied to pixel-space diffusion on ImageNet, Latent Forcing achieves a new state of the art for diffusion transformer-based pixel generation at our compute scale. Code and checkpoints at https://github.com/AlanBaade/LatentForcing

Current image generation work has a tradeoff in how we represent the image. One approach, latent diffusion, suggests that we should compress the input pixels of an image into "latents" that make the image easier for neural networks to model. In the other camp, pixel diffusion suggests that we should directly predict pixels to simplify the pipeline and place all modeling into the generative network. Although pixel diffusion is simpler and end-to-end, latent diffusion is still more commonly used because it improves training efficiency and results in better images. Our work demonstrates that we can obtain the best of both worlds by predicting both latents and pixels, as long as we are careful to order the trajectory of information during generation to reveal latents earlier than pixels. With this insight, we further explore order as a fundamental component of image generative models, helping to explain behavior like the "reconstrcution vs generation tradeoff" and the well-known REPA technique.