FeRA: Frequency-Energy Constrained Routing for Effective Diffusion Adaptation Fine-Tuning

Diffusion models have achieved remarkable success in generative modeling, yet how to effectively adapting large pretrained models to new tasks remains challenging. We revisit the reconstruction behavior of diffusion models during denoising to unveil the underlying frequency–energy mechanism governing this process. Building upon this observation, we propose \textbf{FeRA}, a frequency-driven fine-tuning framework that aligns parameter updates with the intrinsic frequency–energy progression of diffusion. FeRA establishes a comprehensive frequency–energy framework for effective diffusion adaptation fine-tuning, comprising three synergistic components: \textit{(i)} a compact frequency–energy indicator that characterizes the latent’s bandwise energy distribution, \textit{(ii)} a soft frequency router that adaptively fuses multiple frequency-specific adapter experts, and \textit{(iii)} a frequency–energy consistency regularization that stabilizes diffusion optimization and ensures coherent adaptation across bands. Routing operates in both training and inference, with inference-time routing dynamically determined by the latent frequency energy. It integrates seamlessly with adapter-based tuning schemes and generalizes well across diffusion backbones and resolutions. By aligning adaptation with the frequency–energy mechanism, \textbf{FeRA} provides a simple, stable, and compatible paradigm for effective and robust diffusion model adaptation. Codes will be made publicly available.

Diffusion models can generate high-quality images, but adapting these large models to new styles, subjects, or domains is often expensive and unstable. Existing efficient fine-tuning methods usually treat all denoising steps in the same way, even though diffusion models gradually reconstruct images from coarse structures to fine details. In this work, we study this process from a frequency perspective and find that the energy of an image changes systematically across frequency bands during denoising. Based on this observation, we propose FeRA, a lightweight fine-tuning method that adapts the model according to the current frequency-energy state of the latent image. FeRA uses a compact frequency-energy indicator to describe the denoising stage, a soft router to combine several specialized adapters, and a consistency regularization to keep updates aligned with the model’s reconstruction behavior. Our method improves image adaptation and personalization across multiple diffusion backbones while using far fewer trainable parameters than full fine-tuning. This makes diffusion model customization more efficient, stable, and broadly applicable.