ECO: Quantized Training without Full-Precision Master Weights

Quantization has significantly improved the compute and memory efficiency of Large Language Model (LLM) training. However, existing approaches still rely on accumulating their updates in high-precision: concretely, gradient updates must be applied to a high-precision weight buffer, known as $\textit{master weights}$. This buffer introduces substantial memory overhead, particularly for Sparse Mixture of Experts (SMoE) models, where model parameters and optimizer states dominate memory usage. To address this, we introduce the Error-Compensating Optimizer (ECO), which eliminates master weights by applying updates directly to quantized parameters. ECO quantizes weights after each step and carefully injects the resulting quantization error into the optimizer momentum, forming an error-feedback loop with no additional memory. We prove that, under standard assumptions and a decaying learning rate, ECO converges to a constant-radius neighborhood of the optimum, while naive master-weight removal can incur an error that is inversely proportional to the learning rate. We show empirical results for pretraining small Transformers (30--800M), a Gemma-3 1B model, and a 2.1B parameter Sparse MoE model with FP8 quantization, and fine-tuning DeepSeek-MoE-16B in INT4 precision. Throughout, ECO matches baselines with master weights up to near-lossless accuracy, significantly shifting the static memory vs validation loss Pareto frontier.

Training today's largest AI models is enormously expensive, and a major reason is memory. Modern hardware runs much faster when numbers are stored in a compact, low-precision format, but stable training currently requires keeping a hidden high-precision copy of every parameter, called master weights, because the tiny adjustments made each step would otherwise be lost in the compact format. These master weights eat up much of the memory savings that low-precision training was supposed to provide. We introduce ECO, a method that removes master weights entirely. The idea is simple: every time the parameters are squeezed back into the compact format, a small amount of information is lost; instead of discarding it, ECO feeds it back into the optimizer so the missing updates are applied a little later. This happens through a buffer the optimizer already maintains, so ECO adds no extra memory. We prove that ECO converges to nearly the same solution quality as training with master weights, whereas naively dropping them causes errors to grow unboundedly. Empirically, across models from 30 million to 16 billion parameters, including large Mixture-of-Experts architectures, ECO matches the accuracy of high-precision training while reducing memory use by up to 25 percent, and it can be combined with other memory-saving techniques for further gains.