Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning

Data unlearning aims to remove the influence of specific training samples from a trained model. In fine-tuning approaches, data unlearning largely relies on loss maximization over forget samples and often suffers from quality degradation or incomplete forgetting. Existing approaches apply unlearning uniformly across diffusion stages, ignoring diffusion dynamics from noise to data. Our systematic study based on diffusion phases shows that forgetting in diffusion models occurs unevenly across time and frequency. By selectively controlling time and frequency, we achieve both higher unlearning success rates and improved generation quality across diverse settings, including both conditional and unconditional scenarios. We also introduce an improved SSCD metric, which measures the dissimilarity in a normalized perturbation distance. Together, our analysis and methods provide practical insights and strategies to improve both evaluation and unlearning performance in diffusion models.