Enhancing LLM Training via Spectral Clipping

While spectral-based optimizers like Muon operate directly on the spectrum of updates, standard adaptive methods such as AdamW do not account for the spectral structure of weights and gradients, leaving them vulnerable to two empirical issues in large language model (LLM) training: (i) the optimizer updates can have large spectral norms, potentially destabilizing training and degrading generalization; (ii) stochastic gradient noise can exhibit sparse spectral spikes, with a few dominant singular values much larger than the rest. We propose SPECTRA, a general framework addressing these by (i) post-spectral clipping of updates to enforce spectral-norm constraints (ii) optional pre-spectral clipping of gradients to suppress spectral noise spikes. We prove that post-clipping constitutes a Composite Frank-Wolfe method with spectral-norm constraints and weight regularization. We further analyze how pre-clipping mitigates sparse spectral spikes. We propose efficient soft spectral clipping via Newton-Schulz iterations, avoiding expensive SVD. Experiments on LLM pretraining show SPECTRA uniformly improves validation loss for various optimizers, including AdamW, Signum, Mars, and AdEMAMix, with the best-performing variants achieving state-of-the-art results. Models trained with SPECTRA exhibit smaller weight norms, confirming the link between spectral clipping and regularization.

Training a large language model means repeatedly adjusting billions of internal numbers until the model gets good at predicting text. Each adjustment is computed by an "optimizer." We found that popular optimizers have two blind spots. First, some adjustments are far too large in certain directions, which can make training unstable and hurt the final model. Second, the random noise in each step occasionally produces a few outsized errors that overwhelm the useful signal. To address these, we built SPECTRA, a simple add-on that trims these adjustments down to a safe size and filters out the noisy spikes, while keeping the genuinely useful information intact. It does this efficiently, without the slow mathematical operations such fixes usually require. When we added SPECTRA to several widely used optimizers and trained language models of varying sizes, it consistently produced better models and allowed faster, more stable training. Since it improves quality without extra memory or meaningful slowdown, SPECTRA can make training large models cheaper and more reliable — lowering the cost and hardware barrier for AI research.