Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoE

Test-time scaling improves LLM performance by generating multiple candidate solutions, yet token-level sampling requires temperature tuning that trades off diversity against stability. Fine-grained MoE, featuring hundreds of well-trained experts per layer and multi-expert activation per token, offers an unexplored alternative through its rich routing space. We empirically characterize fine-grained MoE routing and uncover an informative pattern: router scores exhibit a certain head of high-confidence experts followed by an uncertain tail of low-confidence candidates. While single-run greedy accuracy remains stable when fewer experts are activated, multi-sample pass@n degrades significantly—suggesting that the certain head governs core reasoning capability while the uncertain tail correlates with reasoning diversity. Motivated by these findings, we propose Expert-Sample, a training-free method that preserves high-confidence selections while injecting controlled stochasticity into the uncertain tail, enabling diverse generation without destabilizing outputs. Evaluated on multiple fine-grained MoE models across math, knowledge reasoning, and code tasks, Expert-Sample consistently improves pass@n and verification-based accuracy. On Qwen3-30B-A3B-Instruct evaluated on GPQA-Diamond with 32 parallel samples, pass@32 rises from 85.4% to 91.9%, and accuracy improves from 59.1% to 62.6% with Best-of-N verification.

We present Expert-Sample, a simple yet effective method that introduces diversity at the expert routing level in fine-grained MoE models. Unlike traditional token-level sampling, which faces an inherent trade-off between diversity and stability, Expert-Sample injects randomness earlier in the computation—at the expert selection stage, and remains compatible with normal-temperature decoding. Extensive experiments across four fine-grained MoE models and diverse reasoning tasks demonstrate that Expert-Sample consistently improves pass@n accuracy and provides actual accuracy gains when combined with verification methods. Our work reveals that fine-grained MoE architectures possess untapped potential for inference-time scaling through expert-level interventions. We hope this work inspires further exploration of this complementary dimension to token-level strategies, and provides insights for the training side.