Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles

Diffusion models have demonstrated remarkable generative capabilities and have also emerged as powerful self-supervised representation learners. However, the connection between these two abilities remains less explored. In this work, drawing inspiration from self-supervised learning (SSL), we introduce a framework for jointly evaluating the representation and generation capabilities of diffusion models. Specifically, we decompose features into invariant and residual components and derive the Invariant Contamination Ratio (ICR), a Fisher-based metric that quantifies how residual variation contaminates invariant signal in feature space. We use this framework to analyze both discriminative and generative behavior of diffusion models. On the representation side, we find that invariance peaks at intermediate noise levels, which also yield the best downstream classification performance. On the generative side, we study how training transitions from genuine generalization to memorization in data-limited regimes, and show that ICR serves as a sensitive training-time indicator of early learning: increasing residual energy along Fisher directions marks the onset of memorization, detectable from training features alone without external evaluators or held-out test sets. Overall, our results show that diffusion models can be monitored from a self-supervised perspective through the geometry of their learned representations.

Diffusion models are best known for generating realistic images, but they also learn internal representations that can be useful for other tasks. In this work, we study the connection between these two abilities by analyzing the representations learned inside diffusion models. We find that the most useful representations emerge at intermediate noise levels: too little noise causes the model to focus on low-level details, while too much noise destroys important information. We also study what happens when diffusion models are trained with limited data. In this setting, models can eventually memorize training examples instead of learning general patterns. Our method can detect the onset of this memorization directly from the model's internal representations, without requiring labeled data or held-out test sets. Overall, our work shows that learned representations provide a useful lens for understanding both the capabilities and limitations of diffusion models.