Veda: Scalable Video Diffusion via Distilled Sparse Attention

Scaling Diffusion Transformers to generate high-resolution, long videos is constrained by the quadratic cost of self-attention, and existing sparse attention methods degrade under high sparsity. We show empirically that generation quality is determined not by the sparsity ratio itself, but by how well the sparse mask aligns with the tile-wise geometry of full attention. Based on this insight, we propose Veda, a distilled sparse attention framework that formulates tile selection as an explicit reconstruction problem from full attention. Veda integrates statistics-aware tile scoring with head-aware tiling to reduce estimation error and structural mismatch, enabling aggressive sparsity. A hardware-efficient tile-skipping kernel converts theoretical sparsity into practical wall-clock speedups. Experiments on large video diffusion models, including Waver and Wan2.1, demonstrate substantial acceleration with no noticeable degradation in generation quality. To generate 720P 10-second videos on Waver-T2V-12B, Veda achieves a 5.1$\times$ end-to-end speedup and a 10.5$\times$ self-attention speedup, reducing attention overhead from 92\% to 50\%. Notably, the gains increase with sequence length, indicating that Veda scales favorably with spatiotemporal resolution across models.

Video diffusion models have rapidly advanced toward producing visually faithful videos with rich detail, coherent motion, and increasingly realistic scene dynamics. This progress, however, exposes a fundamental scaling challenge: as videos become longer and higher in resolution, generation requires coordinating far more information across space and time, making the process prohibitively slow. Veda addresses this bottleneck by reducing redundant computation while preserving the essential internal structure that the original model relies on. Rather than simply removing computation, Veda learns which interactions are structurally important and keeps those pathways, allowing the model to spend computation where it matters most. This leads to substantial practical acceleration without noticeable degradation in visual quality. For a 720p, 10-second video with 241 frames, Veda reduces generation time from 19.4 minutes to 3.8 minutes, achieving a 5.1× end-to-end speedup. Since the benefit increases as videos become longer and higher-resolution, Veda provides a concrete path toward practical high-resolution, long-duration video generation.