Diffusion Models Preferentially Memorize Prototypical Examples or: Why Does My Diffusion Model Love Slop?

Generative models have a persistent limitation: their tendency to memorize training data can create legal liabilities and erode creative diversity. Understanding which samples are memorized in whole or in part, and under what conditions, therefore remains an important open problem. Here we answer the question "Are atypical or rare samples memorized first?" in the negative. We train diffusion models on strings generated according to the production rules of the Random Hierarchy Model (RHM), and find that samples composed of common substrings are preferentially memorized. This holds true even if the training data consists of entirely unique samples, indicating that deduplication at the data point level does not provide a meaningful privacy guarantee. Correspondingly we predict, then observe, delayed memorization for fat-tailed datasets (i.e., those with more atypical samples). This effect is amplified when fat-tails are introduced into high-level production rules. These together suggest that dataset diversity, particularly at higher levels of abstraction, plays an important role in staving off memorization. Finally, we identify an intermediate regime of partial memorization in which common substrings are learned first and subsequently overproduced during generation. If training is stopped in this regime, models will exhibit the reversion-to-the-mean blandness often derided as "slop".

Generative AI models, such as large language models like ChatGPT or diffusion models like Veo, can reproduce strings of memorized text or even entire samples observed during training, an undesirable outcome for both creative and legal reasons. Memorization comes with rote repetition: a model that initially produces novel outputs will, after training for too long on stale data, eventually start outputting memorized samples. Yet, this process is poorly understood: During training, what samples are memorized first? Are the first memorized samples atypical outliers or composed of common motifs? Does the overall dataset distribution matter? To study this, we trained diffusion models on synthetic data that aims to capture the hierarchical and compositional structure of real data (images are composed of objects, objects of smaller parts, etc.). We find preferential memorization: models first memorize samples made from common sub-parts, before those with rarer sub-parts. Moreover, before complete sample level memorization, we identify an intermediate training stage of partial memorization characterized by collage: models memorize and incorporate common sub-parts in their outputs. In this phase (which we verify also afflicts image generating diffusion models) output diversity is compromised; outputs composed of common and familiar patterns are overrepresented, leading to generic outputs sometimes criticized as AI slop. As a mitigation strategy, we find that increased dataset diversity reduces memorization and suggest tracking measures of abstract output diversity.