Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers

While Diffusion Transformers (DiTs) have achieved notable progress in video generation, this long-sequence generation task remains constrained by the quadratic complexity inherent to self-attention mechanisms, creating significant barriers to practical deployment. Although sparse attention methods attempt to address this challenge, existing approaches either rely on oversimplified static patterns or require computationally expensive sampling operations to achieve dynamic sparsity, resulting in inaccurate pattern predictions and degraded generation quality. To overcome these limitations, we propose a \underline{\textbf{M}}ixture-\underline{\textbf{O}}f-\underline{\textbf{D}}istribution \textbf{DiT} (\textbf{MOD-DiT}), a novel sampling-free dynamic attention framework that accurately models evolving attention patterns through a two-stage process. First, MOD-DiT leverages prior information from early denoising steps and adopts a {distributed mixing approach} to model an efficient linear approximation model, which is then used to predict mask patterns for a specific denoising interval. Second, an online block masking strategy dynamically applies these predicted masks while maintaining historical sparsity information, eliminating the need for repetitive sampling operations. Extensive evaluations demonstrate consistent acceleration and quality improvements across multiple benchmarks and model architectures, validating MOD-DiT's effectiveness for efficient, high-quality video generation while overcoming the computational limitations of traditional sparse attention approaches.

Generating high-quality videos with AI requires processing enormous sequences of data, where the core computational bottleneck lies in the self-attention mechanism — a component that scales quadratically with sequence length, making long video generation prohibitively expensive. Although sparse attention methods have been proposed to reduce this cost, existing approaches either apply fixed, oversimplified patterns that fail to reflect the true structure of video attention, or rely on costly sampling procedures to achieve adaptivity, resulting in degraded generation quality. This paper presents MOD-DiT (Mixture-Of-Distribution Diffusion Transformer), a novel framework that addresses these limitations through two key insights. First, we identify that attention maps in video diffusion transformers are not arbitrary — they consistently manifest as a dynamic superposition of three structured patterns: block-diagonal (capturing spatial coherence within frames), parallel-to-main-diagonal (encoding temporal continuity across frames), and vertical (representing global semantic dependencies). Second, we show that the intensities of these patterns evolve in a predictable, piecewise linear fashion during the denoising process, enabling accurate forward prediction without any sampling overhead. Building on these observations, MOD-DiT employs a lightweight linear approximation model to reconstruct attention structure and predict sparse masks for future denoising steps entirely online, requiring no model retraining. A hardware-optimized least-squares kernel accelerates this process by over 100× compared to standard solvers, and an iterative fusion strategy continuously corrects prediction drift. Evaluated on state-of-the-art video generation models — including HunyuanVideo and Wan2.1 — MOD-DiT achieves up to 2.05× inference speedup while maintaining video quality nearly indistinguishable from full-attention baselines, as confirmed by comprehensive benchmarks on VBench. These results demonstrate that accurately modeling the mixture of attention distributions is both theoretically principled and practically effective for efficient, high-fidelity video generation.