MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models

Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully. Hence, the trade-off between accuracy and computation in an MoE model typically exhibits large discontinuities. We propose Mixture of Slimmable Experts (MoSE), an MoE architecture in which each expert has a nested, slimmable structure that can be executed at variable widths. This enables conditional computation not only over which experts are activated but also over how much of each expert is utilized. Consequently, a single pretrained MoSE model can support a more continuous spectrum of accuracy-compute trade-offs at inference time. We present a simple and stable training recipe for slimmable experts under sparse routing, combining multi-width training with standard MoE objectives. During inference, we explore strategies for runtime width determination, including a lightweight test-time training mechanism that learns how to map router confidence/probabilities to expert widths under a fixed budget. Experiments on GPT-style models, various routing regimes, zero-shot downstream reasoning benchmarks, and continual pre-training adaptation of DeepSeek model show that MoSE matches or improves standard MoE at full width and consistently shifts the compute-quality frontier toward lower inference FLOPs. The code can be found at: https://github.com/tnurbek/mose.

Large language models are useful, but they can be costly to run. Some models already save compute by choosing only a few specialist parts for each word, but those chosen parts still run at full size every time. We propose a model that can also adjust how much of each chosen specialist part is used. If a part seems less important, the model can use a smaller version of it. If it seems more important, it can use a larger one. This gives a single model many different speed-versus-quality settings. We also show how to choose these settings automatically at test time with very little extra work. In experiments, our method keeps the best full-model quality and often reaches similar or better results with much less computation. This makes language models more efficient and flexible for real-world use, especially when computing resources or response-time limits change.