Spectrally-Guided Diffusion Noise Schedules

Denoising diffusion models are widely used for high-quality image and video generation. Their performance depend on noise schedules, which define the distribution of noise levels applied during training and the sequence of noise levels traversed during sampling. Noise schedules are typically handcrafted and require manual tuning across different resolutions. In this work, we propose a principled way to design per-image noise schedules for pixel diffusion, based on the images spectral properties. By deriving theoretical bounds on how efficacy of minimum and maximum noise levels, we design "tight" noise schedules that eliminate redundant steps. During inference, we propose to conditionally sampled such noise schedules. Experiments show that our noise schedules improve generative quality, particularly at the low-step regime.

Diffusion models are the most popular way to generate images from a text prompt. They start with an image that is pure noise (like the static in older TVs), and gradually transform it into the final image following the prompt. This gradual process requires many steps which makes it slow. We propose a way to reduce the number of steps by using the spectrum of each image, which is easily computed and represent properties like the presence of highly detailed regions (for example the fur on a cat) or large regions of the same color (for example the blue sky). This makes training more effective and improves both the final image quality and generation speed.