Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization

Group Relative Policy Optimization has emerged as essential for aligning video diffusion models with human preferences, but faces a critical computational bottleneck: training a 14B parametered model typically demands hundreds of GPU days per experiment. Existing efficiency methods reduce costs through sliding window subsampling training timesteps, but fundamentally compromise optimization, exhibiting severe instability and failing to reach full trajectory performance. We present Flash-GRPO, a single-step training framework that outperforms full trajectory training in alignment quality under low computational budgets while substantially improving training efficiency. Flash-GRPO addresses two critical challenges: iso-temporal grouping eliminates timestep-confounded variance by enforcing prompt-wise temporal consistency, decoupling policy performance from timestep difficulty; temporal gradient rectification neutralizes the time-dependent scaling factor that causes vastly inconsistent gradient magnitudes across timesteps. Experiments on 1.3B to 14B parameter models validate Flash-GRPO's effectiveness, demonstrating substantial training acceleration with consistent stability and state-of-the-art alignment quality.

Large video generation models can create impressive videos, but making their outputs better match human preferences is still very expensive. Training a large model for this purpose can require hundreds of GPU days for a single experiment, which makes research and development slow and costly. This paper introduces Flash-GRPO, a more efficient training method for improving video generation models. Instead of using a long and expensive training process over many steps, Flash-GRPO learns from a single carefully chosen step while still preserving useful training signals. The method also reduces unwanted variation caused by comparing videos generated at different stages of the generation process, and it balances the strength of training updates so that learning remains stable. Experiments on video models ranging from 1.3 billion to 14 billion parameters show that Flash-GRPO can train much faster than previous approaches while producing videos that better align with human preferences.