Generalization and Scaling Laws for Mixture-of-ExpertsTransformers

We develop a theory of generalization and scaling for Mixture-of-Experts (MoE) Transformers that cleanly separates active per-input capacity from routing combinatorics. By conditioning on fixed routing patterns and union-bounding across them, we derive a sup-norm covering-number bound whose metric entropy scales with the active parameter budget and incurs a MoE-specific routing overhead. Combined with a standard ERM analysis for squared loss, this yields a generalization bound under a $d$-dimensional manifold data model and $C^\beta$ targets, showing that approximation and estimation trade off as in dense networks once active parameters are accounted for appropriately. We further prove a constructive approximation theorem for MoE architectures, showing that, under the approximation construction, error can decrease either by scaling active capacity or by increasing the number of experts, depending on the dominant bottleneck. From these results we derive neural scaling laws for model size, data size, and compute-optimal tradeoffs. Overall, our results provide a transparent statistical reference point for reasoning about MoE scaling, clarifying which behaviors are certified by worst-case theory and which must arise from data-dependent routing structure or optimization dynamics.

Large language models are powerful but expensive because they usually activate billions of parameters each time they process text. Mixture-of-Experts models reduce this cost by storing many specialized components, called experts, while using only a few of them for each word. This raises a basic question: when these models improve, is it because they have more total parameters, or because they use more computation on each word? We study this question mathematically for Mixture-of-Experts Transformers. Our analysis shows that, in a conservative worst-case sense, performance is mainly controlled by the amount of active computation used per word, not by the total number of stored parameters. We also identify an extra cost from the routing mechanism, which decides which experts are used. Experiments on language modeling tasks support this picture but also reveal an important practical effect: when the expert pool is large, experts can specialize and improve performance beyond what the worst-case theory predicts. This helps explain when sparse expert models behave like ordinary dense models and when expert specialization becomes important for efficient AI systems.