Video-SVD: Efficient Video Diffusion via Orthogonal Basis Composition

Video Diffusion Transformers (VDiTs) represent the state of the art in video generation but remain constrained by the quadratic complexity of dense self-attention. To address this attention bottleneck, we analyze the pre-softmax matrix ($QK^\top$) and reveal two key properties: (1) video attention exhibits an effective low-dimensional structure with rapid singular-value decay, and (2) real motion induces hybrid spatio-temporal patterns rather than rigid ``spatial vs. temporal'' layouts. Guided by these observations, we propose Video-SVD, a training-free and plug-and-play acceleration method that does not modify the original network parameters. Video-SVD learns checkpoint-adaptive orthogonal bases offline and, at inference time, replaces expensive dense attention computation with lightweight online subspace projection and basis composition. To preserve high fidelity, Video-SVD further employs layer-shared dual-stream residual modules to recover fine-grained content details and positional information. Across HunyuanVideo and Wan2.1 backbones, Video-SVD achieves significant end-to-end speedups while maintaining high visual quality, reaching 1.92$\times$ on HunyuanVideo, 1.75$\times$ on Wan2.1-1.3B, and 1.79$\times$ on Wan2.1-14B.

Video generation models can create high-quality videos, but they are often very slow and require large amounts of GPU memory because they need to compare many video tokens with each other. This paper proposes Video-SVD, a method that makes video diffusion models faster without changing the original model parameters. The key idea is to find reusable patterns in the attention computation and use them to approximate the expensive part of the model more efficiently. To avoid losing visual quality, the method also adds a lightweight correction module to recover fine details and positional information. Experiments on HunyuanVideo and Wan2.1 show that Video-SVD significantly speeds up video generation while keeping the generated videos visually faithful, especially for complex motion.