SIPO: Stabilized and Improved Preference Optimization for Aligning Diffusion Models

Preference learning has garnered extensive attention as an effective technique for aligning diffusion models with human preferences in visual generation tasks. However, existing alignment approaches such as Diffusion-DPO suffer from two fundamental challenges: training instability caused by high gradient variances at various timesteps and high parameter sensitivities, and off-policy bias arising from the discrepancy between the optimization data and the policy model's distribution. Our first contribution is a systematical analysis of the diffusion trajectories across different timesteps and identify that the instability primarily originates from early timesteps with low importance weights. To address these issues, we propose SIPO, a Stabilized and Improved preference Optimization framework for aligning diffusion models with human preferences. Concretely, a key gradient, \emph{i.e.,} DPO-C&M is introduced to facilitate stabilize training by clipping and masking uninformative timesteps. Followed by a timestep aware importance re-weighting paradigm to fully correct off-policy bias and emphasize informative updates throughout the alignment process. Extensive experiments on various baseline models, including image generation models on SD1.5, SDXL, and video generation models CogVideoX-2B, CogVideoX-5B, and Wan2.1-1.3B, demonstrate that our SIPO consistently promotes stabilized training and outperforms existing alignment methods, with meticulous adjustments on parameters. Overall, these results highlight the importance of timestep-aware alignment and and provide valuable guidelines for improved preference optimization in diffusion models.

Artificial intelligence models that generate images and videos have become incredibly powerful, but they often need extra training to ensure their creations align with what humans actually find appealing or useful. A common way to teach these AI models is by showing them examples of human preferences. However, teaching these models directly can be highly unstable. Because the AI generates images through a step-by-step process, some early steps provide noisy, unhelpful feedback, causing the model's performance to fluctuate or even collapse over time. Furthermore, the preference data used for training often does not perfectly match the AI's current capabilities, creating a mismatch that further confuses the learning process. To solve these issues, we introduce a new framework called SIPO (Stabilized and Improved Preference Optimization). SIPO works by analyzing the step-by-step generation process and automatically masking the noisy, uninformative steps that cause instability. It also dynamically adjusts the importance of the training data so the AI focuses on the most reliable and informative updates. When tested on popular image and video generation models, SIPO consistently produced higher-quality results without breaking down during long training periods. Ultimately, this makes AI generation tools more reliable, consistent, and better matched to what users actually want to create.