TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling

Mixture-of-Experts (MoE) models achieve remarkable performance by sparsely activating specialized experts, yet their massive parameters in experts pose significant challenges for deployment. While low-rank quantization offers a promising route to compress MoE models, existing methods still incur nonnegligible memory overhead and inference latency. To address these limitations, we propose TileQ, a fine-tuning-free post-training quantization (PTQ) method that employs 2D-tiling structured low-rank quantization to share low-rank factors across both input and output dimensions of MoE experts. Furthermore, we introduce an efficient inference technique for TileQ that fuses multiple low-rank expert computations into a single-pass operation, significantly improving hardware utilization. Experiments show that TileQ cuts down additional memory usage up to 10x and reduces inference latency to 5% while preserving state-of-the-art accuracy.

Imagine a highly skilled but very large team of specialists, where only a few members are needed for any given task. While this team is incredibly capable, its sheer size makes it difficult to fit into a standard workspace or to get everyone communicating efficiently. In the world of AI, these "teams" are called Mixture-of-Experts (MoE) models, and their massive size makes them hard to deploy on regular devices. To make these models smaller and faster without losing their intelligence, we developed a new technique called TileQ. Instead of storing every single detail of each specialist, our method identifies and keeps only the most essential "core patterns" that are shared across the team. By doing this, we drastically reduce the amount of memory required—up to 10 times less—while keeping the model just as accurate as the best existing methods. Furthermore, we designed a smarter way to run the calculations, which slashes the processing time to just 5% of the original. This means powerful AI models can now run much more efficiently on everyday hardware, making advanced technology more accessible.