Float8@2bits: Entropy Coding Enables Data-Free Model Compression

Post-training compression is currently divided into two contrasting regimes. On the one hand, fast, data-free, and model-agnostic methods (e.g., NF4 or HQQ) offer maximum accessibility but suffer from functional collapse at extreme bit-rates below 4 bits. On the other hand, techniques leveraging calibration data or extensive recovery training achieve superior fidelity but impose high computational constraints and face uncertain robustness under data distribution shifts. We introduce EntQuant, a framework that unites the advantages of these distinct paradigms. By matching the performance of data-dependent methods with the speed and universality of data-free techniques, EntQuant enables practical utility in the extreme compression regime. Our method decouples numerical precision from storage cost via entropy coding, compressing a 70B parameter model in less than 10 minutes. We demonstrate that EntQuant does not only achieve state-of-the-art results on standard evaluation sets and models, but also retains functional performance on more complex benchmarks with instruction-tuned models, all at modest inference overhead.

Modern AI language models are powerful but enormous, often demanding more memory than a single computer has. To run them privately on your own hardware, rather than through an outside service, people "compress" these models by storing their billions of internal numbers more coarsely. But compressing them enough to fit usually breaks the model, unless you retrain it on its original training data, which is frequently private, legally protected, or simply unavailable. We introduce EntQuant, a method that compresses large models without any data or retraining. Instead of crudely rounding each number to fewer digits, we gently adjust the numbers so that values repeat far more often, then apply a classic lossless compression technique (the same idea behind ZIP files) that shrinks repetitive data efficiently. A fast, specialized routine unpacks the numbers on the fly while the model runs. This lets us shrink a model to as little as an eighth of its size while keeping its accuracy nearly intact, processing a 70-billion-parameter model in under ten minutes. Our approach makes powerful AI cheaper and more practical to run privately, even in sensitive fields like healthcare and finance where data cannot be shared.