GradPower: Powering Gradients for Faster Language Model Pre-Training

We propose **GradPower**, a lightweight gradient-transformation technique for accelerating language model pre-training. Given a gradient vector $\boldsymbol{g}=(g_ {i})_ {i}$, GradPower first applies the elementwise `sign-power` transformation: $ \varphi_ p(\boldsymbol{g}) = \left({\rm sign}(g_ i)|g_ i|^p\right)_ {i} $ for a fixed $p>0$, and then feeds the transformed gradient into a base optimizer. Notably, GradPower requires only a **single-line code change** and no modifications to the base optimizer’s internal logic, including the hyperparameters. When applied to AdamW (termed **AdamWPower**), GradPower consistently achieves lower terminal loss across diverse architectures (LLaMA, Qwen2MoE), parameter scales (66M to 2B), datasets (C4, OpenWebText), and learning-rate schedules (cosine, warmup-stable-decay). The most pronounced gains are observed when training modern mixture-of-experts models with warmup-stable-decay schedules. GradPower also integrates seamlessly with other state-of-the-art optimizers, such as Muon, yielding further improvements. Finally, we provide theoretical analyses that reveal the underlying mechanism of GradPower and highlight the influence of gradient noise.

Training large language models is costly. GradPower is a simple one-line change to the training process that helps models learn more efficiently without redesigning existing systems. Across several model sizes, datasets, and training setups, it achieved better results with almost no added cost, suggesting an easy way to make model training cheaper.