When Diffusion Language Models Hesitate: Detecting and Correcting Visual Hallucinations via Confidence Fluctuation

Multi-modal Diffusion Language Models (MDLMs) have emerged as a powerful alternative to autoregressive models in vision understanding, offering advantages in bidirectional context modeling and parallel decoding. However, existing MDLMs suffer from visual hallucinations due to the static nature of visual perception. Unlike autoregressive models, MDLMs lack the sequential dependency to dynamically interact with visual content. Therefore, MDLMs rely on fixed visual features encoded at initialization, causing the denoising process to drift toward language priors and lose its anchor to visual evidence. In this paper, we propose VGR (Visual-Guided Refinement), a framework that enables MDLMs to revisit visual details by exploiting diffusion dynamics. Our key insight is that the temporal trajectory of confidence during denoising reveals intrinsic uncertainty: while grounded tokens converge smoothly, hallucinated ones exhibit pronounced confidence fluctuation. VGR utilizes this fluctuation signal to detect uncertain spans and corrects them through targeted visual evidence extraction and in-place remasking. Extensive experiments on image captioning and hallucination evaluation benchmarks demonstrate that our method reduces hallucinations and recalls more details. We release our code at https://github.com/SongWZ3214/VGR.

AI systems that describe images in natural language are typically built on autoregressive models, which generate text word by word. A newer alternative, multi-modal diffusion language models, generates an entire description in parallel through iterative denoising, offering better global consistency and faster inference. However, these models tend to "make things up" when describing fine-grained details, because they only look at the image once at the very beginning of generation and never re-examine it. We discovered that these mistakes are not silent. When the model is about to fabricate a detail, its internal confidence in that specific token fluctuates noticeably during generation, even when the final answer appears certain. Building on this hesitation signal, we developed a system that pinpoints the suspicious word, zooms into the corresponding image region, and asks the model to rewrite that part based on a closer look. Our method substantially reduces fabricated details while recovering small visual elements the model would otherwise miss, improving the performance of multi-modal diffusion language models.