SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation

Distribution Matching Distillation (DMD) is a widely used paradigm for accelerating inference in few-step video diffusion models. However, DMD-style training faces a structural bottleneck: the student-side auxiliary score network (the fake score) must closely track a continuously evolving generator. Updating the fake score too frequently increases training cost and can over-emphasize inner-loop tracking, while infrequent updates lead to tracking lag that destabilizes training and degrades generation consistency. To address this issue, we propose \textbf{Score Gradient Matching Distillation (SGMD)}. SGMD adopts a fake-score perspective by directly optimizing the fake score toward the teacher, while using teacher stop-gradient Fisher as a stable distribution-matching objective. We provide a gradient analysis that motivates this objective choice under ideal tracking. Building on this, SGMD introduces a pair of dual potentials: negative-residual (NR) for outer-loop correction and residual-contraction (RC) for inner-loop tracking. Empirically, compared to DMD, SGMD achieves an approximately $\sim 3\times$ training speedup and substantially improves motion dynamics for 4-step distilled models while preserving temporal consistency.

Modern video generation models can create high-quality videos, but they usually require many repeated computation steps, making them slow and expensive to use. A common way to speed them up is to train a smaller or faster version that can generate videos in only a few steps. However, existing training methods for this setting can be costly and may produce videos with weaker motion. We propose SGMD, a new training method for few-step video generation. The key idea is to better coordinate the video generator with an auxiliary model used during training, so that the system can learn efficiently without repeatedly updating this auxiliary model many times. In experiments on a large-scale video generation model, SGMD reduces training cost by about three times and produces videos with stronger motion. Human evaluations also show that people prefer SGMD in overall quality and motion quality, while visual quality and text-video alignment remain comparable.