Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding

The development of long-context Large Language Models (LLMs) is constrained by the memory bandwidth bottleneck and quadratic complexity of the attention mechanism during decoding. To overcome the inherent trade-offs between the memory overhead of metadata-based metrics and the computational inefficiency of adaptive selection strategies, we present \textbf{Faster Flash Decoding (FFD)}, a novel hardware-algorithm co-design framework designed to break the memory wall in long-context decoding. FFD integrates the selector and computer into a fully fused kernel, replacing external metadata indices with content-aware scanning via low-bit quantization. Furthermore, we introduce the top-$\delta$ strategy, which dynamically filters blocks to achieve distribution-adaptive sparsity without global synchronization. As a training-free, plug-and-play solution, FFD enables the reuse of scanning results for computation, achieving up to 11.6x kernel-level speedup at 256k context length and 2.37x end-to-end throughput improvement. Empirical validation on Ruler and Longbench confirms that FFD maintains model accuracy while delivering high-ratio sparsity.

Large language models today can read very long texts, but generating responses word by word becomes extremely slow. The reason is that for each new word, the model needs to scan through everything it has read so far, and moving all that data from memory to the processor becomes the bottleneck. We propose a method called Faster Flash Decoding that greatly speeds up this process. Instead of checking every piece of past information, our approach makes a quick first pass using a heavily compressed version of the stored data to identify which parts are actually relevant, and then only loads those parts for detailed processing. This strategy avoids storing extra indexing information and adapts naturally to different situations. Our method works on existing models without any retraining, achieves up to 11.6 times faster processing, and can handle inputs as long as an entire novel, all while keeping the model's output quality unchanged.