M+Adam: Low-Precision Training via Additive–Multiplicative Optimization

Training with quantized weights can reduce costs but often results in degraded accuracy, especially when optimization is carried out in low precision, without storing high-precision copies. We identify a key failure mode: under low precision, standard optimizers can get stuck and not make progress, especially at large weight magnitudes due to coarse mantissa resolution. To overcome this, multiplicative updates have been previously proposed, in place of additive updates in standard optimizers. While successful under extremely low precision, such as under the logarithmic number system, they suffer from failures near zero and across sign changes. The failure modes of additive and multiplicative updates are therefore complementary. To exploit this, we propose M+Adam, which combines both update types: additive steps handle sign changes and small magnitudes, while multiplicative steps ensure progress at large magnitudes when additive updates are zeroed out under rounding. We prove monotone descent for M+Adam under standard smoothness assumptions. Across LLaMA-style pretraining with 60M--1B models, $1$--$8\times$ Chinchilla budgets, and using only BF16, FP8, and FP4 master weights, M+Adam consistently improves low-precision training.

Training large AI models is expensive since it requires large amounts of memory, computation, and energy. One way to cut costs is to store and update model weights using fewer bits. However, this can make training unreliable: when the weights are too coarse, many small updates become zero after rounding, so the model stops improving and progress is stalled. This paper introduces M+Adam, a new training method designed to make training more reliable in such scenarios. The key idea is to combine two kinds of updates. One kind adds or subtracts small adjustments, which works well for small numbers, while the other kind makes relative or percentage-based changes, which work well when the values are large. By using both together, M+Adam avoids the weaknesses of either approach alone. Experiments on language model training show that M+Adam improves training quality across several model sizes and increasingly fewer bits to represent model weights. These results suggest that better training rules can help make large AI models cheaper and more stable to train without relying on expensive storage of model weights.