PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning

As large language models (LLMs) continue to scale, it becomes increasingly challenging to grow model capacity under fixed computation budgets. We propose Path-Aligned Decompression Distillation (PADD), a framework for distilling knowledge from dense teachers without explicit routing into mixture-of-experts (MoE) students while learning high-quality routing policies. PADD organizes knowledge distillation into four stages in two phases: an initialization phase (Stage I) that builds diverse functionality in the student's experts through teacher neuron clustering and student-expert warmup, and a training phase (Stages II--IV) that integrates online adaptive distillation, path-refined policy optimization, and reward-augmented load balancing in a single training pipeline.Experiments on mathematical reasoning benchmarks demonstrate that PADD yields substantial gains over strong baselines at the same inference cost and that the MoE student can match or surpass its dense teacher. They also demonstrate effective teacher-to-student knowledge distillation and stable routing behavior.

Large language models keep growing, but training and running ever-larger “all-on” models is expensive. A popular alternative is a mixture-of-experts design: many smaller specialist modules, with only a few activated per input—like a team where a dispatcher sends each question to the right experts. The catch is that most strong teachers today are dense models with no dispatcher at all, so it is unclear how to transfer their skills into a mixture-of-experts student without the student’s routing collapsing into noise early on. We developed PADD (Path-Aligned Decompression Distillation), a step-by-step training recipe that first groups the teacher’s internal “neuron teams” to initialize student experts, then trains the student along its actual routing paths using adaptive teacher feedback, stabilized policy learning, and load balancing that rewards useful experts—not just busy ones. On challenging math reasoning benchmarks, PADD improves over strong baselines at the same inference cost, and the mixture-of-experts student can match or beat its dense teacher. This makes it more practical to build capable, efficient models by upgrading from proven dense teachers rather than training huge mixture-of-experts systems from scratch.