ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns

Mixture-of-Experts (MoE) scales model capacity while preserving computational efficiency through sparse expert activation. However, training high-quality MoEs from scratch is prohibitively expensive. An alternative is to convert pretrained dense models into sparse MoEs. Existing dense-to-MoE methods fall into two categories: \textbf{dynamic structural pruning} that converts dense models into MoEs with moderate sparsity to balance performance and efficiency, and \textbf{downcycling} approaches that use pretrained dense models to initialize highly sparse MoEs. However, existing methods break the intrinsic activation patterns within dense models, leading to suboptimal expert construction. In this work, we argue that the Gated Linear Unit (GLU) provides a natural blueprint for dense-to-MoE conversion. We show that the fine-grained neuron-wise activation patterns of GLU reveal a coarse-grained structure, uncovering an inherent MoE architecture composed of consistently activated universal neurons and dynamically activated specialized neurons. Leveraging this discovery, we introduce ExpertWeaver, a training-free framework that partitions neurons according to their activation patterns and constructs shared experts and specialized routed experts with layer-adaptive configurations. Experiments demonstrate that ExpertWeaver outperforms existing methods, both as a training-free dynamic structural pruning technique and as a downcycling strategy for MoE initialization.

Large language models are powerful but expensive to train and run, which limits who can use or improve them. This paper studies how to make existing large models more efficient without training them from scratch. We find that these models already contain useful internal patterns showing which parts are commonly needed and which parts are only useful for specific types of inputs. Based on this observation, we propose ExpertWeaver, a method that reorganizes an existing model into smaller functional groups so that only the most relevant parts need to be used for each input. This can reduce computation while preserving much of the model's ability. Our experiments show that this approach works well both for making models cheaper to use and for creating strong starting points for further training.