Robustness of Mixtures of Experts to Feature Noise

Despite their practical success, it remains unclear why Mixture of Experts (MoE) models can outperform dense networks beyond sheer parameter scaling. We study an iso-parameter regime where inputs exhibit latent modular structure but are corrupted by feature noise, a proxy for noisy internal activations. We show that sparse expert activation acts as a noise filter: compared to a dense estimator, MoEs achieve lower generalization error under feature noise, improved robustness to perturbations, and faster convergence speed. Empirical results on synthetic data and real-world language tasks corroborate the theoretical insights, demonstrating consistent robustness and efficiency gains from sparse modular computation.

(1) Problem: Large language models are becoming incredibly powerful, but their massive size requires huge amounts of computing power and energy. To fix this, researchers use a clever design called "Mixture of Experts" (MoE), which only activates a small, specialized part of the model for any given task. While we know MoEs are efficient, it has remained a mystery why they often perform better than standard models of the exact same size. (2) Solution: We discovered that the secret lies in how MoEs handle internal "noise." Imagine trying to listen to a single conversation in a crowded, noisy room. A standard model tries to process all the noise at once, which leads to confusion. In contrast, an MoE acts like a highly focused listener, activating only the relevant "expert" for the task and completely ignoring the irrelevant background noise. We proved this noise-filtering ability both mathematically and through experiments on real-world language models. (3) Impact: Our findings provide a fundamental understanding of why MoEs are so robust and quick to learn. This insight will guide the design of future artificial intelligence systems that are not only smarter and more reliable, but also far cheaper and environmentally friendly to run.