XPERT: Expert Knowledge Transfer for Effective Training of Language Models

Mixture-of-Experts (MoE) language models organize knowledge into explicitly routed expert modules, making expert-level representations traceable and analyzable. By analyzing expert activation patterns in MoE language large models (LLMs), we find that a subset of experts is consistently activated across diverse knowledge domains. These common experts encode cross-domain, generalizable knowledge that is closely related to model generalization, naturally raising the question of how such identifiable expert knowledge can be practically reused. Motivated by this observation, we propose XPERT, a training-free framework that extracts, consolidates, and reuses expert knowledge from pre-trained MoE LLMs to support effective training of language models across different model scales. XPERT identifies cross-domain experts via inference-only analysis, refines their representations through tensor decomposition, and adapts the extracted knowledge to be reused in downstream models. Experiments on language understanding and dialogue generation benchmarks show that models benefiting from reused expert knowledge achieve consistently stronger performance and faster convergence compared to strong baselines. These results highlight MoE LLMs as structured and reusable knowledge sources, and demonstrate the value of expert-level knowledge reuse for improving model training.

Training language models from scratch is expensive and often requires large amounts of data. This paper explores a way to reuse useful knowledge from existing Mixture-of-Experts language models. We observe that some experts are repeatedly activated across many different domains, which suggests that they store general knowledge that can benefit other models. We propose XPERT, a framework that extracts and consolidates this expert knowledge and uses it to initialize new language models. Across language understanding and dialogue generation tasks, XPERT helps models learn faster and perform better than strong baselines. Our findings show that expert knowledge in large language models can be reused in a practical way to make model training more efficient.