PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

Mixture-of-Experts (MoE) have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input. However, their deployment remains limited due to the high memory overhead associated with storing all expert parameters, particularly as the number of experts increases. To address this challenge, prior works have explored expert dropping and merging strategies; however, they often suffer from notable performance drop especially at high compression ratios due to their reliance on coarse-grained tensor- or expert-level operations. In this paper, we introduce PuzzleMoE, the first MoE merging method to enable fine-grained element-wise merging while achieving both high accuracy and inference speed, via two key innovations: First, PuzzleMoE performs sparse expert merging by identifying element-wise weight redundancy and specialization. It introduces a dual-mask approach to capture both shared and expert-specific salient parameters. Second, to avoid the overhead of storing masks and signs, we introduce a bit-packed encoding scheme that reuses underutilized exponent bits, enabling efficient MoE inference on GPUs. Extensive experiments demonstrate that PuzzleMoE outperforms prior MoE compression methods by up to 16.7\% on MMLU at 50\% compression ratio, and achieves up to 1.80$\times$ end-to-end inference throughput gain.

Modern AI systems increasingly rely on Mixture-of-Experts (MoE) LLMs, which contain many specialized expert components. These models can deliver strong performance while using only a small subset of experts for each request. However, all experts still need to be stored in memory, making MoE models expensive to deploy on real-world hardware. Existing compression methods often remove or merge experts in a coarse way, which can significantly reduce model quality. In this work, we introduce PuzzleMoE, a new method for compressing MoE models while preserving their performance. Instead of treating each expert as a single unit, PuzzleMoE identifies which individual parameters are shared across experts and which ones are important for each expert’s unique knowledge. It then selectively merges only the redundant parts while keeping critical information intact. To further improve efficiency, we design a compact encoding technique that stores the compressed model without additional metadata overhead and supports fast execution on modern consumer GPUs. Across multiple state-of-the-art MoE language models and evaluation benchmarks, PuzzleMoE consistently outperforms previous compression methods, enabling the practical deployment of large MoE LLMs on consumer GPUs while reserving models' performance. Our approach reduces memory requirements, improves inference speed, and helps make large AI models more accessible for deployment on resource-constrained hardware.