ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling

While Mixture-of-Experts (MoE) architectures substantially bolster the expressive power of large-language models, their prohibitive memory footprint severely impedes the practical deployment on resource-constrained edge devices, especially when model behavior must be preserved without relying on lossy quantization. In this paper, we present ZipMoE, an efficient and semantically lossless on-device MoE serving system. ZipMoE exploits the synergy between the hardware properties of edge devices and the statistical redundancy inherent to MoE parameters via a caching-scheduling co-design with provable performance guarantee. Fundamentally, our design shifts the paradigm of on-device MoE inference from an I/O-bound bottleneck to a compute-centric workflow that enables efficient parallelization. We implement a prototype of ZipMoE and conduct extensive experiments on representative edge computing platforms using popular open-source MoE models and real-world workloads. Our evaluation reveals that ZipMoE achieves up to 72.77% inference latency reduction and up to $6.76\times$ higher throughput than the state-of-the-art systems. Our code is available at: https://github.com/npnothard/ZipMoE-ICML26.

Mixture-of-Experts (MoE) is a class of incredibly powerful LLMs. However, running them on everyday devices like phones or laptops has been a long-standing headache, usually requiring developers to downgrade the model and sacrifice its accuracy. We asked a simple question: Can we bring fast MoE inference directly to your mobile devices without compromising performance at all? Our paper gives an affirmative answer to this question by introducing ZipMoE, a new solution to losslessly accelerate MoE inference on edge platforms. Exploiting the observation that LLM parameters contain significant redundancy from an information-theoretic view, we use losslessly compressed parameters during inference. This transforms expert access from loading data across congested PCIe bandwidth into an expert reconstruction process that can be computed in parallel using multiple CPU cores. ZipMoE is the very first MoE inference engine for edge platforms utilizing lossless compression, demonstrating order-of-magnitude improvements in latency and throughput. Our solution opens up the possibility of a broad range of private, responsive, and always-available on-device LLM applications.