How Excess Latent Dimensionality Delays Memorization in Diffusion Models

We study how the gap between latent and intrinsic dimensionality shapes memorization in diffusion models. In practice, the latent dimensionality of diffusion models does not match the dataset's true intrinsic dimensionality. Extending recent spectral analysis to the regime where latent dimensionality is greater than the intrinsic dimensionality, we find that the excess null dimensions populate a new "noise-dimension'' bulk in the score network's feature-correlation spectrum, sitting between population-driven and sample-specific eigenmodes. Because the network learns modes in order of decreasing eigenvalue, this bulk acts as a buffer: each excess latent dimension adds a mode that must be absorbed before memorization can begin, increasing the gap between generalization and memorization. We further find that at very large latent dimensions, trainable score networks partially recover from the increase in score error that the frozen-feature theory predicts, consistent with the first layer learning sparse, signal-aligned features. These findings are supported by experiments on real and synthetic datasets and a random-feature analysis.