Abstract: We obtain an analytic, interpretable and predictive theory of creativity in convolutional diffusion models. Indeed, score-matching diffusion models can generate highly original images that lie far from their training data. However, optimal score-matching theory suggests that these models should only be able to produce memorized training examples. To reconcile this theory-experiment gap, we identify two simple inductive biases, locality and equivariance, that: (1) induce a form of combinatorial creativity by preventing optimal score-matching; (2) result in fully analytic, completely mechanistically interpretable, local score (LS) and equivariant local score (ELS) machines that, (3) after calibrating a single time-dependent hyperparameter can quantitatively predict the outputs of trained convolution only diffusion models (like ResNets and UNets) with high accuracy (median $r^2$ of $0.95, 0.94, 0.94, 0.96$ for our top model on CIFAR10, FashionMNIST, MNIST, and CelebA). Our model reveals a {\it locally consistent patch mosaic} mechanism of creativity, in which diffusion models create exponentially many novel images by mixing and matching different local training set patches at different scales and image locations. Our theory also partially predicts the outputs of pre-trained self-attention enabled UNets (median $r^2 \sim 0.77$ on CIFAR10), revealing an intriguing role for attention in carving out semantic coherence from local patch mosaics.

Lay Summary:

Modern generative AI is capable of producing a seemingly unlimited amount of apparently "creative" output, showing the capacity to mix and match features from the data it was trained on in novel and often unpredictable ways. Understanding how this process occurs, and how the outputs that these models produce relates to the task that they were trained to performed and the data that they learned from, is a key question for understanding the nature of artificial intelligence. To understand where this "creativity" emerges from, we decided to study the simplest models that we could find that exhibited this ability, called a "convolutional diffusion model." We developed a mathematical theory to explain their behavior, based on a handful of properties that they exhibit. This theory predicted that, while "smarter" models might be able to recall their training data, these simple models could only "mix and match" bits and pieces of the dataset at a time-- forming "patchwork quilts" of all of the images that they had ever seen in their training set. While seemingly far-fetched, this theory was remarkably predictive, and in fact, we were able to reproduce almost exactly the images that they produced, directly from their training data-- a first in the field of generative AI. Our theory also explained why AI makes certain common mistakes when generating images, such as putting in incorrect numbers of limbs.