Grouter: Decoupling Routing from Representation for Accelerated MoE Training

Traditional Mixture-of-Experts (MoE) training typically proceeds without any structural priors, effectively requiring the model to simultaneously train expert weights while searching for an optimal routing policy within a vast combinatorial space. This entanglement often leads to sluggish convergence and training instabilities. This paper introduces Grouter a preemptive routing method that by distilling high-quality structures from fully-trained MoE models and serving as a fixed router for target models. By decoupling structural optimization from weight updates, Grouter significantly accelerates both the speed and quality of model convergence. To ensure the framework's versatility, we also introduce expert folding to adapt Grouter across varying model configurations and expert tuning to rebalance workloads across different data distributions. Furthermore, by leveraging the structural priors provided by preemptive routing, we can implement targeted optimizations to further enhance training throughput. Experiments demonstrate that Grouter achieves superior performance and efficiency which boosts pre-training data utilization by $\textbf{4.28$\times$}$ and achieves up to $\textbf{33.5$\%$}$ throughput acceleration, establishing preemptive routing as a fundamental paradigm for scalable MoE training. We publicly release our code and pretrained Grouter checkpoints.

Training large AI models efficiently is one of the biggest challenges in modern machine learning. A popular approach called Mixture-of-Experts (MoE) builds models with many specialist components, activating only a few per input to save computation. However, during training, the system must simultaneously learn which specialists to assign to which inputs and how each specialist should behave — two tasks that constantly interfere with each other, causing slow and unstable training. We solve this by learning the assignment strategy once from an already-trained model, then providing it as a fixed "roadmap" before training begins. This way, each specialist receives consistent, relevant data from the start and can focus entirely on improving its skills. Our method reduces the data needed to reach a given performance level by over 4×, while also accelerating training speed by up to 33.5%. We release our code and pre-trained routing models publicly.