Understanding MARS: When Scaling Momentum Provably Helps

MARS (Yuan et al., 2025) has recently emerged as a strong optimizer for large language model (LLM) training by scaling the correction term in momentum-based variance reduction (MVR). However, existing theory does not explain why this modification can improve convergence over the unscaled MVR choice $\gamma=1$. In this paper, we provide a theoretical explanation for this phenomenon. We introduce **$\gamma$-similarity**, a refined similarity condition that captures how the scaling coefficient interacts with the stochastic gradient-difference structure. This condition recovers standard similarity at $\gamma=1$ and smoothness at $\gamma=0$. Using $\gamma$-similarity, we derive convergence guarantees for fixed-$\gamma$ MARS whose complexity depends explicitly on $\gamma$ and the corresponding $\gamma$-similarity constant. The bound reveals why small values of $\gamma$ can be beneficial: they may reduce the similarity term enough to outweigh the penalty from deviating from MVR. We prove that optimizing $\gamma$ gives MARS a lower complexity guarantee than MVR. Experiments with MARS-AdamW on GPT-style LLM pretraining corroborate the theory, showing that properly chosen small values of $\gamma$ improve token efficiency over $\gamma=1$ and AdamW under a fixed training protocol.

Training large AI models requires an optimizer, a rule for updating the model from noisy information about the data. MARS is a recent optimizer that often works well in language-model training. Its key modification is simple: it scales a correction term that is meant to reduce noise in the update. Before this work, there was no clear theory explaining why making this correction smaller could improve training rather than hurt it. We study the scaling mechanism behind MARS and introduce a new way to measure when the scaled correction is useful. Our theory shows that, in favorable regimes, choosing the scaling coefficient properly can make MARS require fewer gradient computations than the unscaled method. Experiments on language-model pretraining also show that this coefficient matters: small values can train more efficiently than the unscaled choice under the same training setup. Overall, the paper explains why a simple scaling trick can make variance reduction more effective for modern large-model training.