Scaling Beyond Masked Diffusion Language Models

Diffusion language models are a promising alternative to autoregressive models due to their potential for faster generation. Among discrete diffusion approaches, Masked diffusion currently dominates, largely driven by strong perplexity on language modeling benchmarks. In this work, we present the first scaling law study of uniform-state and interpolating discrete diffusion methods. We also show that Masked diffusion models can be made approximately 12% more FLOPs-efficient when trained with a simple cross-entropy objective. We find that perplexity is informative within a diffusion family but can be misleading across families, where models with worse likelihood scaling may be preferable due to faster and more practical sampling, as reflected by the speed-quality Pareto frontier. These results challenge the view that Masked diffusion is categorically the future of diffusion language modeling and that perplexity alone suffices for cross-algorithm comparison. Scaling all methods to 1.7B parameters, we show that uniform-state diffusion remains competitive on likelihood-based benchmarks and outperforms autoregressive and Masked diffusion models on GSM8K, despite worse validation perplexity.

Most large language models write text one token at a time, which can be slow, while diffusion language models can revise many parts of a sentence in parallel. Recent work has mostly focused on one kind of diffusion model, called masked diffusion, and judged progress using perplexity, a score of how well a model predicts text. We asked whether this focus hides better choices for real-world generation. We compared three diffusion approaches under the same training budgets: masked diffusion, uniform-state diffusion, and a hybrid approach that mixes diffusion with left-to-right generation. We also found that a simpler training objective makes masked diffusion about 12% more compute-efficient. The main finding is that the model with the best prediction score is not always the best model to use: some alternatives generate text faster at similar quality because they need fewer refinement steps or can reuse previous computation. When scaled to 1.7B parameters, the uniform-state model stayed competitive on standard language benchmarks and did best on a math reasoning task after fine-tuning, despite having worse perplexity. These results suggest that future diffusion language models should be evaluated by the combined trade-off between quality, speed, and downstream usefulness, not by perplexity alone.