Simple Denoising Diffusion Language Models

Recent Uniform-state Diffusion Models (USDMs), initialized from a uniform prior, offer the promise of fast text generation due to their inherent self-correction ability compared to masked diffusion models. However, they still rely on complex loss formulations with additional computational overhead, which hinders scalability. In this work, we explore a simplified denoising-based loss for USDMs that optimizes only noise-replaced tokens, stabilizing training while matching the performance of prior methods with more complex objectives. In addition, we introduce an efficient regularization term to mitigate corruption toward uniform output distributions, which further improves performance. We demonstrate the effectiveness and efficiency of our simple and improved loss formulations by pretraining models on widely used text datasets for USDMs. More importantly, our conclusions scale to larger models, showing strong potential for large-scale training.

Modern language models usually generate text one word at a time, which can be slow and difficult to revise once mistakes are made. Diffusion language models offer a different approach: they start from corrupted text and gradually refine it into meaningful sentences, making it possible to correct earlier errors during generation. Among them, Uniform-state Diffusion Models (USDMs) start from random uniformly sampled tokens and have shown promise for faster and more flexible text generation. However, existing training methods for these models often use complicated objectives that add extra computational cost, making them harder to scale to large models and datasets. In this work, we show that USDMs can be trained with a much simpler denoising objective: the model only needs to learn to recover the tokens that were replaced by noise. This simple design makes training more stable and efficient while preserving the performance of previous, more complex methods. We also introduce a lightweight regularization technique that prevents the model from producing overly uniform and uninformative predictions. Experiments on standard text datasets show that our method improves training efficiency and generation quality, and the benefits remain when scaling to larger models. Our results suggest a practical path toward scalable USDMs.