TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization

Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. Ternarization has emerged as a promising compression technique, offering significant reductions in model size and inference complexity. However, existing methods struggle with heavy-tailed activation distributions and therefore keep activations in high precision, fundamentally limiting end-to-end inference acceleration. To overcome this limitation, we propose TWLA, a post-training quantization (PTQ) framework that achieves 1.58-bit weight compression and 4-bit activation quantization while maintaining high accuracy. TWLA comprises three components: (1) Euclidean-to-Manifold Asymmetric Ternary Quantizer (E2M-ATQ) minimizes layer-output error under weight ternarization via a two-stage optimization from Euclidean initialization to manifold relocation; (2) Kronecker Orthogonal Tri-Modal Shaping (KOTMS) applies a Kronecker-structured orthogonal rotation to reshape weights into ternary-friendly tri-modal distributions, while the shared rotation statistically suppresses activation outliers; and (3) Inter-Layer Aware Activation Mixed Precision (ILA-AMP) explicitly introduces adjacent-layer second-order interaction costs in bit allocation and jointly optimizes for the layer-wise disparity of activation quantization gains induced by the shared orthogonal transform, preventing cascades triggered by a few weak layers. Extensive experiments demonstrate that TWLA is a PTQ method that maintains high accuracy under the W1.58A4 configuration, while delivering significant inference acceleration. The code is available at TWLA.

Large Language Models (LLMs) are incredibly capable, but they require massive amounts of memory and computing power, making them expensive and hard to run on everyday devices. To fix this, researchers try to "compress" these models by reducing the precision of their internal numbers. While we can successfully compress the model's static memory (weights) into just three simple values (-1, 0, +1), doing the same for the dynamic data flowing through the model (activations) usually destroys its accuracy due to unpredictable, extreme number spikes. To solve this, we introduce TWLA, a new compression framework. TWLA mathematically reshapes the model's weights to perfectly fit the three-value system while simultaneously smoothing out the extreme spikes in the dynamic data. It also carefully distributes the exact amount of precision needed across different parts of the model to prevent any single weak link from breaking the system. Ultimately, TWLA allows massive AI models to run significantly faster and use far less memory without losing their intelligence. This breakthrough makes it cheaper and more practical to deploy advanced AI on resource-limited devices, bringing powerful AI tools closer to everyday use.