WaterSIC: Information-Theoretically (Near) Optimal \\Linear Layer Quantization

This paper considers the problem of converting a given dense linear layer to low precision. The tradeoff between compressed length and output discrepancy is analyzed information theoretically (IT). It is shown that a popular GPTQ algorithm may have an arbitrarily large gap to the IT limit. To alleviate this problem, a novel algorithm, termed "WaterSIC", is proposed and is shown to be within a rate gap of 0.255 bits to the IT limit, uniformly over all possible covariance matrices of input activations. The key innovation of WaterSIC's is to allocate different quantization rates to different columns (in-features) of the weight matrix, mimicking the classical IT solution known as ``waterfilling''. Applying WaterSIC to the Llama and Qwen family of LLMs establishes new state-of-the-art performance for all quantization rates from 1 to 4 bits. Our code is available at https://github.com/egorlifar/watersic.

Large language models are useful but expensive to store and run, because their weights usually take many bits to represent. We study how to compress the weights of these models while changing their outputs as little as possible. Our method, WaterSIC, gives more bits to the parts of a layer that matter more and fewer bits to the parts that matter less, similar to how water naturally fills the lowest parts of a container first. This idea comes from information theory and leads to a simple change to common quantization methods. We prove that, under a standard mathematical model, WaterSIC is very close to the best possible compression rule. In experiments on Llama and Qwen language models, WaterSIC gives better accuracy than prior post-training quantization methods at the same model size, especially at low bit rates. This makes it easier to store and deploy strong language models on limited hardware, while also giving a clearer theoretical understanding of what good model compression should look like.