A Random Matrix Perspective on the Consistency of Diffusion Models

Diffusion models trained on different, non-overlapping subsets of a dataset often produce strikingly similar outputs when given the same noise seed. We trace this consistency to a simple linear effect: the shared Gaussian statistics across splits already predict much of the generated images. To formalize this, we develop a random matrix theory (RMT) framework that quantifies how finite datasets shape the expectation and variance of the learned denoiser and sampling map in the linear setting. For expectations, sampling variability acts as a renormalization of the noise level through a self-consistent relation $\sigma^2\to\kappa(\sigma^2)$, explaining why limited data overshrink low-variance directions and pull samples toward the dataset mean. For fluctuations, our variance formulas reveal three key factors behind cross-split disagreement: \textit{anisotropy} across eigenmodes, \textit{inhomogeneity} across inputs, and overall scaling with dataset size. Extending deterministic-equivalence tools to fractional matrix powers further allows us to analyze entire sampling trajectories. The theory sharply predicts the behavior of linear diffusion models, and we validate its predictions on UNet and DiT architectures in their non-memorization regime, identifying where and how samples deviates across training data split. This provides a principled baseline for reproducibility in diffusion training, linking spectral properties of data to the stability of generative outputs.

Diffusion models generate images by gradually turning random noise into structured samples, so one might expect two separately trained models to behave very differently. Surprisingly, models trained on different subsets of the same dataset can often produce very similar images when started from the same noise. We study why this happens. Our work shows that much of this consistency can be explained by simple Gaussian statistics shared across data splits. In a mathematically tractable version of diffusion models, the generated image is largely determined by the dataset’s average structure and its main directions of variation, such as common variations in faces or natural images. We use tools from random matrix theory to predict how independently sampled data splits lead to deviations in these Gaussian statistics, and how those deviations propagate to the diffusion model and its generated samples. The theory predicts not only why independently trained models agree, but also where they disagree: differences concentrate along particular visual directions, are amplified in specific regions of the initial noise space, and shrink as the dataset grows. This provides a clear analytical baseline for understanding reproducibility, memorization, and generalization in modern generative models.