Consistent Diffusion Language Models

Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps. In continuous domains, consistency training along the probability-flow ODE is a popular recipe to accelerate diffusion. For discrete diffusion, no analogous sample-space ODE exists, making direct adaptation ill-defined. We argue that the right discrete substitute is the exact posterior bridge—the closed-form conditional law linking any two noise levels—which is available for broad corruptions including masked and uniform diffusion. Building on this observation, we introduce Multi-Path Discrete Consistency (MPDC), a new principle that trains a denoiser to be path-invariant in expectation across these stochastic bridges, and instantiate it as the Consistent Diffusion Language Model (CDLM), a single-stage training framework that does not require an already trained teacher model. Our CDLM objective recovers masked diffusion, continuous consistency models, and progressive or discrete distillation as analytic limits or empirical approximations of one common view. Empirically, CDLM establishes a new state of the art on both conditional and unconditional text-generation, consistently outperforming strong base discrete diffusion models and often even multi-stage distilled baselines across sampling budgets, with the largest gains in the few-step regime. Together, these results position CDLM as a principled and scalable foundation for the next generation of fast, high-fidelity discrete generative modeling.

Most AI chatbots can be thought of as writing text one word at a time, from left to right. This works well, but it can be slow because each new word depends on the words that came before it. Diffusion language models offer a different approach: start with a complete but corrupted piece of text and gradually clean it up, allowing many parts of a sentence to improve at the same time. In principle, this could make text generation much faster, but in practice these models often need hundreds of clean-up passes to produce high-quality writing, largely wiping out the speed advantage. For image generation, researchers have tackled a similar problem by teaching models to skip ahead along a single, well-defined clean-up path. Text is harder, since there is no single correct path from corrupted text back to a finished sentence, because many different intermediate versions are possible. Our key insight is that, even without one fixed path, the mathematics of diffusion gives an exact way to connect different levels of corruption. We train the model so that taking one large clean-up jump agrees with taking several smaller ones. As a result, the model learns to make much bigger jumps while preserving quality. The resulting system generates high-quality text in just a handful of passes instead of hundreds, often matching or outperforming existing accelerated methods while being trained from scratch. In our experiments, it achieves up to 32× faster generation, making diffusion-based language models more practical, affordable, and energy-efficient.