DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1$\times$ end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.

Modern AI systems can generate realistic videos, but producing each video remains computationally expensive, especially for longer and higher-resolution videos. A major reason is that these systems examine relationships among many small parts of a video across both space and time. Existing methods try to reduce this cost by skipping less important interactions, but they often hurt video quality because the important information is scattered, fine-grained, and changes throughout the generation process. We study when these shortcuts succeed or fail, and use this analysis to design DFSAttn, a method that preserves the most crucial interactions while avoiding much of the unnecessary computation. DFSAttn reorganizes video pieces so that scattered important information becomes easier for GPUs to process, estimates which groups of interactions matter most, and reuses sparse patterns when it is safe to do so. It does not require retraining the video model. DFSAttn maintains high video quality while making video generation up to 2.1 $\times$ faster, helping make high-quality video generation more efficient and accessible.