Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference

The quadratic cost of attention limits the scalability of long-context LLMs, especially under limited hardware memory budgets. While attention is often sparse, existing static sparse methods cannot adapt to task- or input-dependent variations, and recent dynamic approaches rely on predefined templates or heuristics that may sacrifice generality. We propose Dynamic Hierarchical Sparse Attention (DHSA), a data-driven framework that predicts attention sparsity online while keeping the LLM backbone frozen. DHSA performs hierarchical routing by estimating importance at the chunk level and propagating it to token-level interactions, preserving causally important dependencies while enabling efficient sparsification. Across Needle-in-a-Haystack test, LongBench and RULER, DHSA maintains near-dense accuracy in highly sparse regimes, achieving 12--20% relative accuracy gains over Block Sparse Attention at comparable prefill cost. With a memory-efficient tiled backend, DHSA delivers up to $10\times$ prefill speedup at 128K context length. On LLaMA-3.1-8B (4-bit), DHSA scales to 100K context on a single 24GB GPU, where dense attention fails. We provide complementary GPU and CPU backends, enabling DHSA to run across diverse hardware environments and multiple open-weight model families. These results demonstrate DHSA as an efficient and adaptable solution for memory-constrained long-context LLM inference. Code is available at: https://github.com/xiongsiheng/DHSA.

Large language models can now process long documents, conversations, and code files, but doing so requires a large amount of computation and memory. This makes long-context use difficult on limited hardware, such as a single consumer GPU or smaller deployment environments. Our work introduces Dynamic Hierarchical Sparse Attention (DHSA), a method that helps a language model focus on the most relevant parts of a long input instead of processing every possible token interaction. The method first divides the input into meaningful chunks, then identifies which chunks and tokens are most important for the current computation. This allows the model to skip many less useful computations while keeping the original language model unchanged. Across several long-context benchmarks, DHSA keeps accuracy close to full attention while substantially reducing the cost of processing long inputs. It can handle much longer contexts under the same memory limits, including 100K-token inputs on a single 24GB GPU where standard full attention runs out of memory. These results suggest that DHSA can make long-context language models more practical and efficient for real-world applications.