SD-MoE: Spectral Decomposition for Effective Expert Specialization

Mixture-of-Experts (MoE) architectures scale Large Language Models via expert specialization induced by conditional computation. In practice, however, expert specialization often fails: some experts become functionally similar, while others functioning as de facto shared experts, limiting the effective capacity and model performance. In this work, we analysis from a spectral perspective on parameter and gradient spaces, uncover that (1) experts share highly overlapping dominant spectral components in their parameters, (2) dominant gradient subspaces are strongly aligned across experts, driven by ubiquitous low-rank structure in human corpus, and (3) gating mechanisms preferentially route inputs along these dominant directions, further limiting specialization. To address this, we propose Spectral-Decoupled MoE (SD-MoE), which decomposes both parameter and gradient in the spectral space. SD-MoE improves performance across downstream tasks, enables effective expert specialization, incurring minimal additional computation, and can be seamlessly integrated into a wide range of existing MoE architectures, including Qwen and DeepSeek.

Large language models are often made more efficient by dividing their internal computation among multiple “experts,” so that different parts of the model handle different kinds of information. However, in practice these experts often fail to become truly different from one another: many learn similar patterns, and some are used almost all the time. This reduces the benefit of using an expert-based model. In this paper, we study why this happens by looking at the main directions in which expert weights and training updates change. We find that different experts often share the same dominant directions, and that the routing mechanism tends to send inputs based on these shared features rather than on expert-specific behavior. To address this, we propose SD-MoE, a method that separates shared information from expert-specific information during both model initialization and training. Experiments on Qwen- and DeepSeek-based models show that SD-MoE improves expert specialization, improves downstream task performance, and adds only a small amount of training cost while keeping inference cost unchanged.