Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models

Diffusion Language Models (DLMs) have recently achieved significant success due to their any-order generation capabilities. However, existing inference methods typically rely on local, immediate-step metrics—such as confidence or entropy—which inherently lack a more reliable perspective, leading to sub-optimal generation quality. To address this, we propose Coherent Contextual Decoding (CCD), a novel inference framework built upon two core innovations. First, CCD bypasses the potential bias of the single context to leverage historical contexts for approximating the marginal distribution of token prediction, leading to better sequence coherence and the early rejection of sub-optimal paths. More importantly, we demonstrate that this mechanism is theoretically equivalent to modeling the consistency of historical steps via the conditional mutual information between contexts and token predictions. Finally, CCD achieves significantly milder performance degradation under highly parallel decoding scenarios compared to baselines. Empirically, our method achieves a simultaneous enhancement in both inference speed and performance across diverse benchmarks on Dream and LLaDA.

We're teaching AI language models a new way to write text. Unlike traditional models that generate words one after another from left to right, a newer type called Diffusion Language Models can fill in words in any order — much like solving a crossword puzzle. However, current methods decide which word to fill in next based only on a quick, single guess, which often leads to choices that look good in the moment but don't fit well with the rest of the sentence. We developed a smarter approach called Coherent Contextual Decoding (CCD). Instead of relying on just one snapshot, CCD looks back at a short history of previous guesses to make more reliable, consistent decisions about which words to commit to. We also show mathematically why this approach works better. In our experiments, CCD made these models both faster and more accurate across a wide range of tasks, including math and coding problems.