Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models

Discrete diffusion language models (dLLMs) provide a fast and flexible alternative to autoregressive models (ARMs) via iterative denoising with parallel updates. However, their evaluation is challenging: existing metrics conflate denoiser approximation error with sampler-induced error from the sampling dynamics, a problem that does not arise for ARMs whose autoregressive sampling exactly reflects the learned probability model. We introduce a sampler-centric oracle framework that replaces learned denoisers with an exact Hidden Markov Model posterior derived from a ground-truth Markov chain, isolating sampler-induced error in a controlled setting. We show that few-step discrete diffusion samplers are not distributionally correct even under an oracle denoiser, with transition-level mismatch that vanishes only as the number of steps approaches the sequence length. Moreover, improvements in negative log-likelihood (NLL), generative perplexity (GenPPL), or MAUVE do not imply correct sampling. Code is available at https://luhantang.github.io/dllm_sampler/.

Discrete diffusion language models (dLLMs) generate text by repeatedly filling in many missing words in parallel, instead of producing one word at a time like traditional autoregressive language models. This can make generation faster and more flexible, but it also raises an important question: is the sampling procedure itself actually producing text from the distribution it is supposed to follow? In this work, we start from the observation that dLLMs have two key components: a learned denoiser that predicts missing tokens, and a sampler that decides how to use those predictions to generate a full sequence. Existing evaluations usually mix together errors from these two components. To separate them, we build a controlled “oracle” setting where the correct predictions are known exactly using a simple Markov-chain model and Hidden Markov Model inference. We then test several discrete diffusion samplers using this exact oracle, so any remaining mistakes must come from the sampler rather than the learned model. Our results show that few-step diffusion samplers can still fail to follow the intended distribution, even when given perfect predictions. We also find that common evaluation scores, such as perplexity-style metrics and MAUVE, can look good while missing important sampling errors. More broadly, our findings help move toward the goal of dLLM samplers that are not only fast, but also verifiably faithful to the distribution they are meant to sample from.