Less Token, More Signal: MoE Expert Pruning via Critical Token Selection

Mixture-of-Experts (MoE) architectures provide strong scalability for large language models, but their large expert parameter footprint poses challenges for efficient deployment. Expert pruning is widely used to reduce model size and inference cost; however, existing approaches are token-agnostic, treating all tokens equally when estimating expert importance. This uniform treatment dilutes the contributions of informative tokens and leads to suboptimal pruning decisions. To address this fundamental limitation, we propose **Step** (**S**elective **T**oken-guided **E**xpert **P**runing), a token-aware framework that rethinks expert pruning from the perspective of selective token guidance. By incorporating loss-aware expert evaluation and a lightweight knowledge-preserving mechanism, **Step** reduces information loss while removing redundant experts. Extensive experiments across different MoE architectures and model scales demonstrate the effectiveness of **Step**. On the 30B Qwen3 MoE model with 50\% expert sparsity, **Step** achieves nearly a 50\% reduction in memory usage with minimal performance degradation, delivers a 1.5$\times$ throughput improvement, and completes the entire pruning process within 10 minutes.

We wanted to make large AI models more efficient to deploy. These models often use a mixture-of-experts designs, which help them handle complex tasks but require huge amounts of memory, making them costly to run. A common technique to shrink these models is expert pruning, where less useful parts are removed. However, current pruning methods treat every input token the same way, which can accidentally remove important experts and reduce model accuracy. To solve this, we created **Step**, a new pruning approach that focuses on the most informative tokens. **Step** evaluates experts based on their contribution to the model's performance and preserves essential knowledge during pruning. This ensures that only redundant experts are removed while minimizing information loss. Our experiments show that **Step** greatly reduces memory usage and speeds up processing with little impact on quality. For example, on a large model with half its experts pruned, **Step** cut memory use by nearly 50\%, increased throughput by 1.5$\times$, and completed the pruning process in just 10 minutes. This makes AI models more practical for real-world applications.