Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Experts

As the computational demands for pre-training Large Language Models (LLMs) continue to surge, the need for efficient training paradigms becomes critical. Despite the vast resources already invested in existing pre-trained checkpoints, these assets often remain under-leveraged due to architectural limitations. We introduce an "orthogonal growth" strategy designed to "recycle" these checkpoints by strategically expanding their parameters prior to continued training. Our method focuses on optimizing converged Mixture-of-Experts (MoE) models through two dimensions: interpositional layer copying for increased depth and noisy expert duplication for expanded width. Through extensive scaling laws analysis, we demonstrate a strong positive correlation between the "sunk cost" (prior investment) and the final model accuracy. Empirical results on models up to 70B parameters and 1T tokens show that our recycling approach yields a 10.6\% accuracy improvement compared to training from scratch under identical extra compute budgets. This work provides a cost-effective blueprint for sustainable large-scale LLM development.

Training large AI models from scratch is extremely expensive — it can cost millions of dollars in computing resources. A natural question is: can we start with a smaller, already-trained model and grow it into a larger one, reusing the work already invested? We show that this is possible for a class of AI models called Mixture-of-Experts (MoE), which power many modern AI systems. These models route different inputs to different specialized sub-networks ("experts"), achieving high capacity without proportionally increasing computation. We develop two complementary ways to grow such models: making them deeper (adding more processing layers) and making them wider (adding more experts). Crucially, we demonstrate that these two growth directions are largely independent, because they modify different parts of the model and don't interfere with each other, so they can be combined freely in sequence. We validate our approach at two scales. First, we grow a 3-billion-parameter model through 6B to 12B to study the method in detail. Then, we scale up to a real-world setting: starting from a 17-billion-parameter model trained on 600 billion words, we grow it to 35B and then 70B over one trillion words total. The final 70B model outperforms a same-cost model trained from scratch by over 10%, demonstrating that the computational investment in the smaller model is effectively recycled rather than wasted. This challenges the common assumption that scaling up means starting over.