StreamAttention: Energy-Efficient and High-Utilization Attention on Systolic Hardware

Attention computation represents the dominant cost in modern transformers as it grows quadratically with the sequence length. FlashAttention [Dao et al., 2022] cuts the memory cost through tiling and online softmax computation. But the underlying hardware itself remains the bottleneck. Modern accelerators are optimized for matrix multiplication, while the non-linear softmax operation is typically offloaded to much lower-throughput vector units, leading to pipeline stalls. We present StreamAttention, an accelerator co-designed with FlashAttention-2 [Dao, 2023] that sustains continuous streaming of operands on a single systolic array of multiply-accumulate (MAC) units. We map online attention to MAC recurrences that fit the systolic dataflow exactly, and evaluate the softmax exponential as a Chebyshev polynomial with the same MAC units. A four-stage pipeline overlaps every attention phase with no idle cycles between tiles, keeping operands continuously streaming through the array. StreamAttention achieves 95–98% utilization against $\sim$40% for the closest peer SystolicAttention [Lin et al., 2025]. We demonstrate that the Chebyshev approximation has negligible impact on Llama-3.2-1B (WikiText-103 Perplexity), ViT-Base (ImageNet Top-1), and BERT-Large (SQuAD F1). Furthermore, fusing softmax onto the array eliminates per-tile SRAM round-trips of the score and softmax matrices, cutting per-layer attention energy by up to $\sim$2.8$\times$ and average attention power by up to $\sim$2.9$\times$ compared to our baseline. This comes at the cost of a $\sim$34% area overhead and +6% power on pure matrix multiplication over a standard systolic-based matrix unit. We synthesize against the open source SkyWater 130 nm process design kit for area and power, and SRAM energies are from CACTI 7 at 90 nm.