Revisiting Spectral Representations in Generative Diffusion Models

Diffusion models have shown remarkable performance on diverse generation tasks. Recent work finds that imposing representation alignment on the hidden states of diffusion networks can both facilitate training convergence and enhance sampling quality, yet the mechanism driving this synergy remains insufficiently understood. In this paper, we investigate the connection between self-supervised spectral representation learning and diffusion generative models through a shared perspective on perturbation kernels. On the diffusion side, samples (e.g., images, videos) are produced by reversing a stochastic noise-injection process specified by Gaussian kernels; on the spectral representation side, spectral embeddings emerge from contrasting positive and negative relations induced by random perturbation kernels. Motivated by this, we propose a self-supervised spectral representation alignment method to facilitate diffusion model training. In addition, we clarify how joint spectral learning can benefit diffusion training from a geometric perspective. Furthermore, we find that the optimization of the spectral alignment objective is in an equivalent form of diffusion score distillation in the representation space. Building on these findings, we integrate a spectral regularizer into diffusion training objectives to improve the performance of diffusion models on multiple datasets. Experiments across images and 3D point clouds show consistent gains in generation quality.

Diffusion models can generate realistic images, shapes, and other data by learning how to reverse a gradual noising process. Recent work has shown that their training can improve when the model's internal features are aligned with useful visual representations, but this often depends on large pretrained encoders that may not exist for specialized data such as 3D shapes or scientific structures. This raises a broader question: can diffusion models learn better internal representations by themselves, without relying on external teacher models? We study a connection between diffusion generation and self-supervised representation learning: both learn from carefully perturbed versions of the same data. Based on this connection, we introduce a self-supervised spectral regularizer that encourages the hidden features of a diffusion model to preserve the local geometric structure of the data as noise changes over time. Intuitively, the model is guided to understand which noisy samples still belong near each other and which ones should be treated as different. We also show that this objective is closely related to score distillation in representation space, giving a principled explanation for why it helps generation. In experiments on image and 3D generation tasks, this approach improves sample quality and training behavior without requiring external pretrained encoders. This work suggests a practical way to make diffusion models learn both how to generate data and how to organize their internal understanding of that data under a unified mechanism.