LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment

We identify the Spectral Energy Gain in extreme model compression, where low-rank binary approximations outperform tiny-rank floating-point baselines for heavy-tailed spectra. However, prior attempts fail to realize this potential, trailing state-of-the-art 1-bit methods. We attribute this degradation to Latent Geometry Misalignment: standard singular vectors exhibit high coherence (spiky distribution), the worst-case geometry for binary quantization. To realize this gain, we propose LittleBit-2, a framework employing Internal Latent Rotation and Joint Iterative Quantization (Joint-ITQ). This approach acts as a geometric preconditioner, aligning coherent latent distributions with the binary hypercube with zero inference overhead. Empirically, LittleBit-2 establishes a new state-of-the-art in the sub-1-bit regime (1$\sim$0.1 bpp) on Llama-2 and Llama-3, matching the fidelity of leading 1-bit baselines.

Modern AI language models are extraordinarily capable, but they are also enormous, often too large to fit on a phone or laptop. Making them smaller usually means storing each internal number using fewer "bits" of information, and the most aggressive methods shrink each number down to a single bit. Our work pushes even further, into the "less-than-one-bit" range, so that large models can run on memory-limited devices. The difficulty is that squeezing this hard normally ruins the model's accuracy. We found that the problem is largely geometric: when the internal information is rounded to single bits, it sits in an awkward orientation that causes large errors. Our method, LittleBit-2, rotates this information into a friendlier arrangement before compressing it, like turning a key so it fits the lock. This rotation is computed once in advance and adds no cost while the model is running. The result is a new state of the art for extreme compression: our tiny models match the accuracy of much larger one-bit models, helping bring powerful AI to devices where it previously could not run.