Rethinking Sparse Mixture of Experts from a Unified Perspective

Sparse Mixture of Experts (SMoE) models scale the capacity of models while maintaining constant computational overhead. SMoE methods fall into two categories: Token Choice, which routes each token to a fixed number of experts, and Expert Choice, which assigns a fixed number of tokens to each expert. However, the use of fixed budgets for tokens or experts causes both approaches to select irrelevant token–expert pairs or overlook critical assignments, which degrades overall performance. To fill that gap, we rethink SMoE from a unified perspective through the lens of linear programming, which provides a general formulation for SMoE models. Furthermore, we introduce Unified Sparse Mixture of Experts (USMoE), a novel framework comprising a unified mechanism and a unified score to overcome these limitations. We provide both theoretical justification and empirical evidence demonstrating USMoE's effectiveness. Extensive evaluations across diverse data settings (clean and corrupted), multiple domains (including texts and vision tasks), and different learning approaches (training-free and training-based) show that USMoE not only delivers significant performance improvements over existing SMoE methods, but also enables more flexible expert selection budgets, reducing inference costs without compromising model performance.

Modern AI systems often use a design called Mixture-of-Experts, where a large model is divided into many smaller subnetworks, known as "experts". For each input, such as a question or an image, the model must decide which experts should process it. This expert-selection process is crucial for making the model both accurate and efficient, but it is also difficult to design and train effectively, especially as models become larger and more complex. Our paper introduces a unified perspective for improving expert selection. Instead of viewing the problem as either assigning inputs to experts or assigning experts to inputs, we formulate it as selecting the best input–expert pairs. This perspective provides a clearer and more flexible way to understand how Mixture-of-Experts models route information, and it helps bridge different existing routing strategies under one framework. The proposed approach is supported by both theoretical analysis and empirical results. Our findings show that better input–expert selection can improve the effectiveness of large-scale AI models while maintaining efficiency. This makes the method useful for researchers and practitioners who want to train or adapt Mixture-of-Experts models on their own datasets and applications.