Why DDIM Hallucinates More Than DDPM: A Theoretical Analysis of Reverse Dynamics

We theoretically study the hallucination phenomena in two canonical diffusion samplers: the stochastic Denoising Diffusion Probabilistic Model (DDPM) and the deterministic Denoising Diffusion Implicit Model (DDIM). We analyze the reverse ODE (DDIM) and SDE (DDPM) for a Gaussian mixture target, proving that after a critical time $\tau$, (a) DDIM can become stuck on the segment connecting the two nearest modes and (b) DDPM *stochasticity* helps it become unstuck from this region, thus avoiding hallucination. Our empirical validation verifies that DDPM has a significantly lower hallucination rate than DDIM when this region is entered. Building on our observations, we exhibit how using additional stochastic steps can help DDIM avoid hallucinations and offer new insights on how to design improved samplers.

Diffusion models are state-of-the-art AI image and video generators, owing to their impressive synthesized samples. Yet, hallucinations, generations that violate structural or semantic constraints, remain a critical problem for these models. For example, a diffusion model may accidentally produce a hand with six fingers, even though a hand with five was requested. We describe when and where hallucinations arise in diffusion models in a simplified setting (i.e., mixture of Gaussians, which is a standard theoretical setting). Specifically, we focus on the types of hallucinations that emerge when the sample lands in between two modes of the distribution. We also examine why introducing randomness in diffusion models helps reduce hallucination rate. This will help to design better, safer diffusion models for more reliable generative AI.