On the Separability of Information in Diffusion Models

Diffusion models transform noise into data by injecting information that was captured in their neural network during the training phase. In this paper we ask: what is this information? We find that, in pixel-space diffusion models, (1) a large fraction of the total information in the neural network is committed to reconstructing small-scale perceptual details of the image, and (2) the correlations between images and their class labels are informed by the semantic content of the images, and are largely agnostic to the low-level details. We argue that these properties are intrinsically tied to the manifold structure of the data itself. Finally, we show that these facts explain the efficacy of classifier-free guidance: the guidance vector amplifies the mutual information between images and conditioning signals early in the generative process, influencing semantic structure, but tapers out as perceptual details are filled in.

A slightly noisy image of a cat still reveals a cat. This is because the small-scale details of the image have little bearing on the meaning of the image. We show that such separation between the 'big picture' and the minutae of images affect how information is stored in diffusion-based image generation models.