Attention Sparsity is Input-Stable: Training-Free Sparse Attention for Video Generation via Offline Sparsity Profiling and Online QK Co-Clustering

Diffusion Transformers (DiTs) achieve strong video generation quality but suffer from high inference cost due to dense 3D attention, leading to the development of sparse attention technologies to improve efficiency. However, existing training-free sparse attention methods in video generation still face two unresolved limitations: ignoring layer heterogeneity in attention pruning and ignoring query-key coupling in block partitioning, which hinder a better quality-speedup trade-off. In this work, we uncover a critical insight that the attention sparsity of each layer is its intrinsic property, with minor effects across different inputs. Motivated by this, we propose SVOO, a training-free Sparse attention framework for fast Video generation via Offline layer-wise sparsity profiling and Online bidirectional co-clustering. Specifically, SVOO adopts a two-stage paradigm: (i) offline layer-wise sensitivity profiling to derive intrinsic per-layer pruning levels, and (ii) online block-wise sparse attention via a novel bidirectional co-clustering algorithm. Extensive experiments on seven widely used video generation models demonstrate that SVOO achieves a superior quality-speedup trade-off over state-of-the-art methods, delivering up to 1.93× speedup while maintaining a PSNR of up to 29 dB on Wan2.1.

Diffusion Transformer models can generate high-quality videos, but they are expensive to run because they compare many visual elements with one another at every step. Our work studies an important question: can these models safely skip some of these comparisons without hurting video quality? We find that the answer is often yes. More importantly, we observe that the amount of computation that can be skipped is mainly determined by each layer of the model itself, and changes only slightly across different input prompts or images. In other words, some layers naturally need dense computation, while others consistently contain more redundant comparisons, regardless of the specific video being generated. Based on this finding, we propose SVOO, a training-free method that first measures how much computation each layer can safely skip, and then uses this information to speed up video generation during inference. Unlike methods that apply the same pruning rule everywhere, SVOO respects the different behaviors of different layers and better preserves video quality. Experiments on widely used video generation models show that SVOO can substantially reduce generation time while maintaining visual fidelity, making high-quality video generation more efficient and practical.