Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density

Mixture-of-Experts (MoE) based Large Language Models (LLMs) have achieved superior performance, yet the massive memory overhead caused by storing multiple expert networks severely hinders their practical deployment. Singular Value Decomposition (SVD)-based compression has emerged as a promising post-training technique; however, most existing methods apply uniform rank allocation or rely solely on static weight properties. This overlooks the substantial heterogeneity in expert utilization observed in MoE models, where frequent routing patterns and intrinsic information density vary significantly across experts. In this work, we propose RFID-MoE, an effective framework for MoE compression by exploiting heterogeneous Routing Frequency and Information Density. We first introduce a fused metric that combines expert activation frequency with effective rank to measure expert importance, adaptively allocating higher ranks to critical expert groups under a fixed budget. Moreover, instead of discarding compression residuals, we reconstruct them via a parameter-efficient sparse projection mechanism to recover lost information with minimal parameter overhead. Extensive experiments on representative MoE LLMs (e.g., Qwen3, DeepSeekMoE) across multiple compression ratios demonstrate that RFID-MoE consistently outperforms state-of-the-art methods like MoBE and $\mathbf{D}^2$-MoE. Notably, RFID-MoE achieves a perplexity of 16.92 on PTB with the Qwen3-30B model at a 60% compression ratio, reducing perplexity by over 8.0 compared to baselines, and improves zero-shot accuracy on HellaSwag by approximately 8%.

Large language models based on Mixture-of-Experts (MoE) architectures can achieve strong performance, but they also require a large amount of memory because they contain many expert networks. This makes them expensive and difficult to deploy in practice. In this work, we propose RFID-MoE, a new compression method that reduces the memory cost of MoE models while maintaining their performance. Our method identifies which expert components are more important based on how frequently they are used and how much information they contain, and allocates model capacity accordingly. We also introduce an efficient way to recover information lost during compression with only a small number of additional parameters. Experiments on several representative MoE language models show that RFID-MoE consistently outperforms existing compression methods, achieving lower memory usage while preserving strong reasoning and language understanding performance. This work may help make large AI systems more efficient and easier to deploy in real-world applications.