Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining

An increasing number of LLMs employ Mixture-of-Experts (MoE) architectures where the feed-forward layer is replaced by a pool of experts and each token only activates a small subset of them. During autoregressive generation, these models often enter a memory-bound regime even for moderate batch sizes because the average expert load grows more slowly than in an equivalent dense feedforward layer. Consequently, MoE latency is governed by the number of activated experts. We introduce a framework for $\textbf{dynamically}$ re-routing token-to-expert mapping to lower this number (and thus, the decode latency) while preserving a comparable quality. Our best results use a $\textbf{batch-aware routing}$ that works by having tokens $\textbf{piggyback}$ experts that have already been loaded into memory due to being crucial to other tokens within the same batch. At batch size $16$, OEA reduces MoE-layer decode latency by $39\\%$ on Qwen3-30B while preserving standard-error-adjusted downstream accuracy, and by $15\\%$ on Qwen3-235B with only small overall degradation on the long-generation benchmark suite.

Large language models are increasingly built from many specialized components, called “experts,” where each word activates only a few experts instead of the whole model. This design can make models more powerful, but during text generation it can also become slow: when many user requests are processed together, the system may need to load many different experts into fast memory, even if each individual word uses only a small number of them. We propose Opportunistic Expert Activation, a method that makes this generation step faster without retraining the model. For each word, our method first keeps the most important experts so that the model still has the computations it most needs. It then lets words reuse additional experts that are already being loaded for other words in the same batch, avoiding extra memory movement. This simple change reduces the number of experts that must be loaded during generation, which lowers latency. On large Qwen3 mixture-of-experts models, our method substantially speeds up the expert layers while largely preserving answer quality, making these models more practical for real-time serving.