RaGEP: Rank-aware Geometric Expert Pruning for Mixture-of-Experts Language Models

Sparse Mixture-of-Experts (MoE) architectures scale model capacity efficiently but suffer from massive static parameter footprints, creating significant deployment burdens on memory-constrained hardware. Existing post-training pruning methods often rely on scalar statistics, ignoring the representational geometry of expert feature spaces. This leads to sub-optimal resource allocation across layers and the retention of redundant experts. To address this, we propose a Rank-aware Geometric Expert Pruning (RaGEP) framework to compress MoE models by analyzing the geometric properties of expert activations. First, in the inter-layer allocation stage, we introduce a Rank-aware budget allocation mechanism that adaptively assigns expert budgets based on the effective rank of layer-wise representations. Second, in the intra-layer selection stage, we propose a Spectral-Salience Pruning metric that harmonizes subspace orthogonality and activation magnitude to identify high-energy orthogonal experts. Extensive experiments across MoE models of different scales show that our method consistently outperforms state-of-the-art baselines on a diverse set of zero-shot tasks, while reducing model size and inference cost.

Modern artificial intelligence models often use a design called "Mixture-of-Experts" (MoE). This architecture divides a large AI system into many smaller "specialist" components, activating only a few of them at any given moment to keep computing fast. However, to run these models, all specialists must still be loaded into the computer's active memory. This creates a massive hardware barrier, making it very expensive to deploy advanced AI on standard computers or everyday devices. To make these models smaller and more affordable, we can "prune" (permanently remove) redundant specialists. Unfortunately, existing methods often decide which specialists to keep based on simple usage statistics. This can accidentally delete rare but critical specialists, or retain multiple specialists that do the exact same job. We introduce a new pruning framework called RaGEP to solve this. RaGEP analyzes the mathematical "shapes" of how these specialists process information. It first figures out which layers of the AI handle more complex concepts and automatically allocates more budget to those layers. Then, it carefully selects a compact team of specialists that are highly distinct from one another, ensuring there is no skill overlap. Our experiments demonstrate that RaGEP substantially reduces model size and memory costs without harming the AI's core reasoning abilities, helping to make powerful AI models accessible on much smaller and cheaper hardware.