From Muon to Gluon: Bridging Theory and Practice of LMO-based Optimizers for LLMs

Recent developments in deep learning optimization have brought about radically new algorithms based on the Linear Minimization Oracle (LMO) framework, such as Muon (Jordan et al., 2024b) and Scion (Pethick et al., 2025b). After over a decade of Adam’s dominance, these LMO-based methods are emerging as viable replacements, offering several practical advantages such as improved memory efficiency, better hyperparameter transferability, and most importantly, superior empirical performance on large-scale tasks, including LLM training. However, a significant gap remains between their practical use and our current theoretical understanding: prior analyses (1) overlook the layer-wise LMO application of these optimizers in practice, and (2) rely on an unrealistic smoothness assumption, leading to impractically small stepsizes. To address both, we propose a new LMO-based framework called Gluon, capturing prior theoretically analyzed methods as special cases, and introduce a new refined generalized smoothness model that captures the layer-wise geometry of neural networks, matches the layer-wise practical implementation of Muon and Scion, and leads to state-of-the-art convergence guarantees. Our experiments with NanoGPT and CNN confirm that our assumption holds along the optimization trajectory, ultimately narrowing the gap between theory and practice.

Training large AI models requires repeatedly adjusting billions of numbers, and the rule used for these adjustments can strongly affect cost and performance. New optimizers such as Muon and Scion have recently shown promising results, but existing theory did not accurately describe how they are actually used: in practice, they update different layers of a neural network separately. We introduce Gluon, a mathematical framework that captures this layer-by-layer structure. We also propose a new way to describe how the training loss changes across layers, recognizing that some layers behave very differently from others. This lets us prove convergence guarantees that better match practical implementations and explains why different layers often need very different learning-rate scales. We test this explanation on language-model and image-classification experiments, including NanoGPT and a CNN, and find that the proposed model closely matches the behavior seen during training. Our results help connect the practical success of modern optimizers with a more realistic theory, offering guidance for designing and tuning future training methods for large neural networks.