Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression

Sub-bit model compression targets storage below one bit per weight; as magnitudes are aggressively compressed, the sign bit becomes a fixed-cost bottleneck. Across Transformers, CNNs, and MLPs, learned sign matrices resist low-rank approximation and are spectrally indistinguishable from an i.i.d. Rademacher baseline. This randomness gives rise to the lower bound of sub-bit model compression—the one-bit wall. Despite this apparent randomness, most weights retain their initialization signs; flips primarily occur via rare near-zero boundary crossings, suggesting that sign-pattern randomness is largely inherited from initialization. We formalize this behavior with sign lock-in theory, a stopping-time analysis of sign flips under SGD noise. Under bounded updates and a rare re-entry condition into a small neighborhood of zero, the number of effective sign flips exhibits a geometric tail. Building on this mechanism, we introduce a from-scratch low-rank sign-template training method that prevents the emergence of this one-bit wall.

One-bit wall—compressing AI models means storing each parameter in as few bits as possible. But at aggressive compression, a hidden barrier appears: every parameter still needs its sign bit (+ or –), which resists compression. Sign lock-in—surprisingly, these signs are not really learned. They are fixed at random initialization and almost never change during training. We theoretically show that this behavior follows from the bounded-update assumption, which is standard in analyses of SGD. Tear down this one-bit wall—we turn sign lock-in from a liability into an advantage. We pick a compressible sign pattern at the start of training and keep it stable. Since the pattern can be regenerated from a tiny seed, the cost of storing the sign drops to nearly zero, allowing models to be compressed beyond previously impossible limits.