SlideSparse: Fast and Flexible (2N-2):2N Structured Sparsity

NVIDIA's 2:4 Sparse Tensor Cores deliver $2\times$ throughput but demand strict 50% pruning—a ratio that causes severe accuracy loss in LLMs. Milder $(2N-2):2N$ patterns (e.g., 6:8, 25% pruning) preserve accuracy far better—within 0.4–1.8 average points of dense in our Qwen2.5-7B/14B study—yet receive *NO* hardware support and fall back to dense execution. We present **SlideSparse**, the first system to unlock Sparse Tensor Core acceleration for the $(2N-2):2N$ model family on commodity GPUs. Our *Sliding Window Decomposition* rewrites any $(2N-2):2N$ weight block into $N-1$ overlapping 2:4-compliant windows without changing the underlying dot product; in addition, our *Activation Lifting* fuses the corresponding activation rearrangement into per-token quantization at low marginal cost. Integrated into vLLM, **SlideSparse** is evaluated across various GPUs (A100, H100, B200, RTX 4090, RTX 5080, DGX-spark), precisions (FP4, INT8, FP8, BF16, FP16), and model families (Llama, Qwen, BitNet). On compute-bound workloads, the measured speedup ($1.33\times$) matches the theoretical upper-bound $N/(N-1)=4/3$ at 6:8 weight sparsity in Qwen2.5-7B, establishing $(2N-2):2N$ as a practical path to better accuracy–speedup trade-offs in LLM acceleration. Code available at https://github.com/bcacdwk/vllmbench.

Running large AI language models is expensive — they require enormous amounts of computation for every response. A common strategy to reduce this cost is "sparsification": setting a portion of the model's numerical values to zero so the hardware can skip unnecessary calculations. Modern GPUs have a built-in fast lane for this, but it only works at one extreme ratio — exactly half the values must be zeroed — which degrades the AI's output quality unacceptably. We found that a milder ratio (zeroing 25% instead of 50%) preserves quality far better. The challenge: GPUs refuse to accelerate it. Our system, SlideSparse, solves this by rearranging the sparsified model into overlapping pieces that each fit the GPU's strict requirements. The rearrangement is mathematically exact — no information is lost. SlideSparse runs on GPUs already deployed in data centers today, with no hardware changes. Across multiple models and chip generations, it delivers up to 1.33× faster AI responses while maintaining output quality — making AI services cheaper and more energy-efficient at scale.