Optimal Stopping in Latent Diffusion Models

We identify and analyze a surprising phenomenon of $\textit{Latent}$ Diffusion Models (LDMs) where the final steps of the diffusion can $\textit{degrade}$ sample quality. In contrast to conventional arguments that justify early stopping for numerical stability, this phenomenon is intrinsic to the dimensionality reduction in LDMs. We provide a principled explanation by analyzing the interaction between latent dimension and stopping time. Under a Gaussian framework with linear autoencoders, we characterize the conditions under which early stopping is needed to minimize the distance between generated and target distributions. More precisely, we show that lower-dimensional representations benefit from earlier termination, whereas higher-dimensional latent spaces require later stopping time. We further establish that the latent dimension interplays with other hyperparameters of the problem such as constraints in the parameters of score matching. Crucially, this framework suggests that the reconstruction quality of the autoencoder alone can serve as a proxy to estimate the potential performance of the full LDM. Experiments on synthetic and real datasets illustrate these properties, underlining that early stopping can improve generative quality. Together, our results offer a theoretical foundation for understanding how the latent dimension influences the sample quality, and highlight stopping time as a key hyperparameter in LDMs.

Diffusion models are a popular family of AI systems used to generate realistic images. They work by gradually removing noise from a random starting point until a clean output emerges. A widely used variant, called Latent Diffusion Models, first compresses data into a smaller representation before applying this noise-removal process, which makes generation faster and cheaper. A common assumption is that running the noise-removal process to completion always produces the best results. We show that this is not the case: for latent diffusion models, the final denoising steps can actually hurt image quality. This happens because the compression step discards some information, and continuing to denoise beyond a certain point amplifies these imperfections rather than correcting them.