General and Efficient Steering of Unconditional Diffusion Models

Steering diffusion models toward conditions unseen during training typically requires either retraining with conditional inputs or per-step gradient computations, both of which incur substantial computational overhead. We present NA-RFM, a general recipe for efficiently steering unconditional diffusion models without gradient guidance during inference, enabling fast controllable generation. The method combines two offline-computed signals: noise alignment, a high-noise correction from PCA statistics of the target examples and the full data, and Recursive Feature Machine (RFM) activation steering, which learns a target-discriminative direction from labeled forward-process activations. During sampling, noise alignment provides coarse control at high noise, while the RFM direction is reused over intermediate/late timesteps through lightweight activation edits. Experiments on CIFAR-10, ImageNet, CelebA, and fine-grained bird species show improved target accuracy over gradient-based post-hoc guidance baselines, improved FID on the class-guidance benchmarks, and substantial inference speedups. Code: https://github.com/isotrivial/na-rfm.

Modern image generators can make realistic pictures, but it is still hard to control a pretrained generator in ways that were not built into its training. For example, a user may have only a few example images of a desired object, attribute, or visual property, rather than a class label or text prompt that the model already understands. Existing post-hoc control methods often steer generation by repeatedly using gradients from an extra classifier, which can be slow. This paper introduces a way to steer a pretrained image generator using example images, without retraining the generator and without gradient computations during sampling. The method first uses simple statistics of the examples to guide the rough structure of the image early in generation. It then learns an internal direction inside the model that is associated with the target examples and adds that direction while the image is being formed. Across several image-generation benchmarks, this approach improves target control compared with gradient-based post-hoc guidance methods while keeping sampling efficient. The results suggest that pretrained generators already contain useful internal directions for new visual targets, and that these directions can be discovered from examples and reused for controllable generation.