A Unified Sparse Attention via Multi-Granularity Compression

Efficient long-context understanding is increasingly vital for large language model (LLM) applications such as multi-turn dialogue and program analysis. However, the core self-attention scales quadratically with sequence length, creating a fundamental computational bottleneck. Existing sparse attention methods alleviate this issue but face trade-offs: training-based methods are costly and cannot be directly applied as acceleration plugins for other models, while inference-time methods often compromise efficiency or cross-modal generality. To address these limitations, we present UniSparse, a unified mechanism that introduces the notion of composite tokens—compact representations that aggregate multi-granularity contextual information. Building on this abstraction, UniSparse dynamically constructs sparse attention through multi-granularity compression and block-level selection, enabling efficient and hardware-friendly execution on GPU. Across multiple modalities and tasks ranging from synthetic benchmarks to real-world applications, UniSparse consistently surpasses state-of-the-art sparse attention methods (e.g., MInference, XAttention, FlexPrefill) in both accuracy and efficiency, achieving $\ge$ 99\% of full-attention accuracy and up to 2.61$\times$ faster attention computation than FlashAttention.

Long-context inference in large language and multimodal models is bottlenecked by self-attention, whose cost grows quadratically with sequence length and dominates latency at scales beyond tens of thousands of tokens. Existing remedies introduce a structural tension: training-coupled sparse architectures embed efficiency into the model weights but cannot accelerate pre-trained models, while inference-time heuristics buy speed at the cost of fidelity and cross-modal generality. The deeper question is whether a single, modality-agnostic abstraction can reconcile both. We propose UniSparse, which reframes sparse-attention selection through a multi-granularity compression lens. Queries and keys are aggregated into composite tokens — compact summaries that compress along the sequence dimension and, optionally, the head dimension. Proxy attention is computed entirely in this reduced space, yielding block-level importance scores that drive a dynamic Top-P selection at orders-of-magnitude lower cost than full attention, while remaining a drop-in replacement for any pre-trained Transformer. Across long-context language and video benchmarks (RULER, HELMET, Video-MME), UniSparse retains ≥99% of full-attention accuracy and delivers up to 2.6× attention speedup over FlashAttention, establishing multi-granularity compression as a unified principle for training-free sparse attention.