Finding DoRI: Discovery of Retained Images in Diffusion Models

Text-to-image diffusion models (DMs) have achieved remarkable success in image generation. However, concerns about data privacy and intellectual property remain due to their potential to inadvertently memorize and replicate training data. Recent mitigation efforts have focused on identifying and pruning weights responsible for triggering verbatim training data replication, based on the assumption that memorization can be localized. We challenge this assumption and demonstrate that, even after such pruning, small perturbations to the text embeddings of previously mitigated prompts can re-trigger data replication, revealing the fragility of such methods. Our further analysis then provides multiple indications that memorization is indeed not inherently local: (1) replication triggers for memorized images are distributed throughout text embedding space; (2) embeddings yielding the same replicated image produce divergent model activations; and (3) different pruning methods identify inconsistent sets of memorization-related weights for the same image. Finally, we show that bypassing the locality assumption enables more robust mitigation through adversarial fine-tuning. These findings provide new insights into the fundamental nature of memorization in text-to-image DMs and inform the future development of more reliable mitigation methods against DM memorization.

AI image generators can generate high-quality images, thanks to a lot of training images. Some of the images end up memorized by the models, which means the model remembers the image pixel-by-pixel and reproduces the image exactly during generation, which is undesired from a copyright perspective. Some methods to remove memorized images try to identify where exactly inside the model the memorized images are stored. After identification, they simply eliminate the memorization by removing specific parts of the model, which then no longer reproduces the memorized content. We show that their success is illusory: the memorized images are still inside the model, and can be recovered using DoRI, our Discovery of Retained Images tool. Next, we show that memorized images are not stored in one place in the model, but instead are spread out everywhere. With that insight, we take a different approach: we specifically train the whole model to fully and successfully remove the memorized images from it.