Rethinking Convergence in MoE Training: The Role of Routing Sparsity

In Mixture-of-Experts (MoE) training, sparse routing, i.e., activating only the top-$K$ experts per token, is essential for balancing convergence speed and computational cost. However, existing works typically choose $K$ empirically, without theoretical guidance. To address this gap, we characterize the convergence behavior of MoE training using stochastic optimization theory. Specifically, we derive a convergence upper bound of $\mathcal{O}\left(\frac{1+M/K}{\sqrt{T}}\right)$, where $T$ is the number of training iterations and $M$ is the total number of experts per MoE layer. This result guarantees convergence and shows that increasing $K$ can accelerate training. By further fixing the total computational budget $R$ (in FLOPs), we obtain a refined bound of $\mathcal{O}\left(\sqrt{\frac{K}{R}} + \frac{M}{\sqrt{K R}}\right)$, which is convex in $K$ and implies the existence of an optimal $K^{*}\in[1,M]$ that achieves the best convergence performance. Extensive experiments validate our theoretical analysis under diverse settings.

Large language models increasingly rely on Mixture-of-Experts (MoE) architectures to scale model capacity while keeping computation manageable. A key component of MoE training is sparse routing, where only a small subset of experts is activated for each input. The number of activated experts has a major impact on training: using more experts can improve learning efficiency but also increases computational cost. In practice, this routing sparsity is usually chosen through trial and error, with little theoretical understanding of how it affects training performance. In this work, we provide a theoretical analysis of how routing sparsity influences the convergence behavior of MoE training. Our results show that activating more experts can accelerate optimization, but only up to a point: under a fixed computational budget, increasing the number of activated experts also raises computation cost, creating an inherent trade-off. We further show that this trade-off implies the existence of an optimal routing sparsity that achieves the best training efficiency. Extensive experiments across diverse settings support our theoretical findings. By providing principled guidance for selecting routing sparsity, our work moves MoE training beyond heuristic tuning and offers insights for building more efficient large-scale AI systems.