Spectral Guidance for Flexible and Efficient Control of Diffusion Models

We introduce Spectral Guidance, a framework for controlling diffusion models by leveraging the intrinsic geometry of the generative process. As data is progressively corrupted by noise, only a small number of features remain informative for control. We characterize them as the singular functions of a conditional expectation operator and show that they can be learned via a self-supervised objective. Once recovered, this basis enables the projection of arbitrary guidance signals, such as labels, CLIP embeddings, or masks, directly onto the sampling trajectory. This approach allows for stable, high-fidelity control without retraining or denoiser backpropagation during sampling. Empirically, we improve conditional accuracy on CIFAR-10 by 37 percentage points over the strongest training-free baseline while offering $4\times$ faster sampling. Moreover, the same representations that support label and CLIP guidance also enable spatial control, such as mask-based guidance, without auxiliary models. Finally, our framework reveals a phase transition in the generative process, pinpointing the optimal time window for effective guidance.

Making AI image generators produce something specific requires "guiding" them. Today this is either costly (retrain an entire system for each new type of control) or fragile (apply quick fixes during generation that often distort the image). We noticed that during generation, the broad strokes of an image emerge first and remain stable, while fine details only fill in at the end. We trained a small companion network to identify these stable broad-stroke features just once. Any guidance signal, from text descriptions to target shapes, can then be mapped onto them and used to steer generation, with no retraining for each new task. Our method matches the requested content more accurately than the best existing guidance techniques while running four times faster. The same learned features handle categories, text prompts, and shape constraints without modification. As a bonus, they reveal a sharp transition during generation that pinpoints when guidance has the most influence. This is useful both as a practical tip and as a window into how these image generators work internally.