Breaking the Factorization Barrier in Diffusion Language Models

Diffusion language models theoretically allow for efficient parallel generation but are practically hindered by the "factorization barrier": the assumption that simultaneously predicted tokens are independent. This limitation forces a trade-off: models must either sacrifice speed by resolving dependencies sequentially or suffer from incoherence due to factorization. We argue that this barrier arises not from limited backbone expressivity, but from a structural misspecification: models are restricted to fully factorized outputs because explicitly parameterizing a joint distribution would require the Transformer to output a prohibitively large number of parameters. We propose Coupled Discrete Diffusion (CoDD), a hybrid framework that breaks this barrier by replacing the fully-factorized output distribution with a lightweight, tractable probabilistic inference layer. This formulation yields a distribution family that is significantly more expressive than standard factorized priors, enabling the modeling of complex joint dependencies, yet remains compact enough to avoid the prohibitive parameter explosion associated with full joint modeling. Empirically, CoDD seamlessly enhances diverse diffusion language model architectures with negligible overhead, matching the reasoning performance of computationally intensive Reinforcement Learning baselines at a fraction of the training cost. Furthermore, it prevents performance collapse in few-step generation, enabling high-quality outputs at significantly reduced latencies. Code available at: https://github.com/liuanji/CoDD

Traditional AI text generators create sentences word-by-word, which is slow. Newer "diffusion langauge models" can generate multiple words at the same time to speed things up, but they hit a major roadblock: they assume that these simultaneously predicted words have no relationship to each other. This forced independence causes the models to generate confusing, jumbled text (like mixing up "New York" and "San Diego" into "San York"). CoDD fixes this structural flaw. Instead of forcing words to be entirely independent, CoDD adds a lightweight, highly efficient layer to the AI's core architexture. This extra layer acts as a smart guide that easily calculates how words naturally depend on one another without overloading the system's memory or speed. CoDD significantly boosts the reasoning and text quality of various AI models while keeping training costs low. Remarkably, CoDD matches the reasoning performance of computationally intensive training alternatives (50 times more expansive) in many-step generation settings, and significantly outperforms them when forced to generate text in fewer steps.