TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control

Large Language Models (LLMs) training is prohibitively expensive, driving interest in low-precision fully-quantized training (FQT). While novel 4-bit formats like NVFP4 offer substantial efficiency gains, achieving near-lossless training at such low precision remains challenging. We introduce **TetraJet-v2**, an end-to-end 4-bit FQT method that leverages NVFP4 for activations, weights and gradients in all linear layers. We identify two critical issues hindering low-precision LLM training: weight oscillation and outliers. To address these, we propose: 1) an unbiased double-block quantization method for NVFP4 linear layers, 2) **OsciReset**, an algorithm to suppress weight oscillation, and 3) **OutControl**, an algorithm to retain outlier accuracy. **TetraJet-v2** outperforms prior methods on FP4 pre-training for LLMs across models up to 370M parameters trained up to 212B tokens, reducing the performance gap to BF16 by an average of $51.3$% while enabling an $1.67\times$ end-to-end speedup over FP8.

Large language models such as chatbots are extremely expensive to train because they require enormous amounts of computation and energy. One promising way to reduce these costs is to use lower-precision arithmetic, where numbers are stored using only 4 bits instead of the much larger formats commonly used today. However, training language models with such tiny numerical representations is difficult: important information can become unstable or distorted, causing the model to lose accuracy. In this work, we introduce TetraJet-v2, a new method for training language models efficiently with 4-bit computations from start to finish. We identify two major problems that make low-precision training unstable: some model weights repeatedly “flip” between nearby values during training, and some activations contain unusually large values that are hard to represent accurately in 4 bits. To address these issues, we design new algorithms that stabilize these oscillating weights and preserve the accuracy of important outlier values. We evaluate our method on language models trained on hundreds of billions of text tokens. Compared to previous 4-bit training methods, TetraJet-v2 produces more accurate models while also running substantially faster. Our approach achieves up to 1.67× training speedup over modern FP8 training, helping make large-scale AI training more affordable and energy-efficient.