LEGO: Expert Expansion for Continual Learning of Mixture-of-Experts Language Models

Mixture-of-experts (MoE)-based large language models (LLMs) scale model capacity without proportional growth in compute, yet adapting them to new domains after pretraining remains challenging. Retraining from scratch is prohibitively expensive, while naive continual learning can cause catastrophic forgetting; i.e., new-domain gradients overwrite previously trained router models in the MoE. To this end, we propose LEGO, a novel expert expansion strategy that adds new experts with target-side routing pressure, steering new-domain tokens toward the expanded capacity, and anchor-side stabilization to prevent weight drift in existing experts relative to the previous checkpoint. The key insight of LEGO is that new-domain learning should be localized in dedicated capacity, leaving existing expert routing undisturbed. Extensive experimental results demonstrate that LEGO consistently improves the trade-off between acquiring new knowledge and preserving prior capabilities.