S-Quant: Rethinking Weight Quantization with Seed-Based Generation

The progressive scaling of large language models (LLMs) has consistently enhanced multimodal understanding and advanced reasoning capabilities, but has substantially increased computational and hardware execution overhead. In this paper, we present S-Quant, a novel post-method that compresses only model weights. We partition each weight tensor into fixed-size blocks and assign a single seed to each block. The seed drives a hardware-friendly Linear Feedback Shift Register (LFSR) generator that dynamically produces multiple basis matrices. Each block is then reconstructed as a linear combination of these basis matrices, with block-specific coefficients, which substantially reduces the amount of stored data, increases the data-transfer efficiency between memory and compute units, and consequently speeds up memory-bound inference for large language models. Experimental results on different LLM models ranging from 7B–70B parameters show that S-Quant attains state-of-the-art performance when weights are compressed to approximately 3-bit or 4-bit. We also design a dedicated ASIC accelerator that achieves a 4× speed-up for memory-bound LLM inference.

Large language models (LLMs) have recently achieved impressive performance in tasks such as reasoning, image understanding, and text generation. However, these advances come at the cost of extremely large model sizes, which require significant memory capacity, data movement, and hardware resources during inference. This makes deploying advanced AI systems expensive and energy intensive, especially on practical hardware platforms. In this work, we propose S-Quant, a lightweight compression method that reduces the storage cost of LLM weights without retraining the model. Instead of storing every weight directly, our method represents groups of weights using compact random seeds and a small set of coefficients. These seeds are used to efficiently regenerate weight patterns during inference through simple hardware-friendly operations. By greatly reducing memory traffic between storage and computing units, S-Quant accelerates inference while maintaining strong model accuracy. Experiments on LLMs ranging from 7B to 70B parameters show that S-Quant achieves state-of-the-art performance at very low bit-widths, while our custom accelerator design further delivers up to 4× faster memory-bound inference.