Scaling depth capacity via zero/one-layer model expansion

Model depth is a double-edged sword in deep learning: deeper models achieve higher accuracy but require higher computational cost. To efficiently train models at scale, progressive training (also known as model expansion) scales up model capacity during training and significantly reduces computation with little performance degradation. In this work, we study the depth expansion of large-scale models through the lens of optimization theory and feature learning, offering insights on the initialization of new layers, hyperparameter transfer, learning rate schedule, and timing of model expansion. Specifically, we propose zero/one-layer progressive training to achieve an optimal tradeoff between computation and loss, with a comprehensive ablations on our expansion strategy. For example, zero/one-layer progressive training on GPT2 can save $\approx 80\%$ compute, or equivalently achieve an $\approx 5\times$ acceleration, while attaining a loss comparable to that of a fully trained 60-layer model with 7B parameters, thus demonstrating a mixing behavior in terms of loss. Furthermore, scaling laws on LLAMA3 and DeepSeekV3 models show a $3\sim 5\times$ improvement in compute efficiency, with an increasing advantage at larger scales.

Modern AI models often become more capable when they are made deeper, but training very deep models from the beginning is expensive. This creates a practical challenge: how can we obtain the benefits of large, deep models without paying the full training cost? We study a simple strategy called progressive training, where a model starts smaller and gains new layers as training proceeds. Our work focuses on when and how to add these new layers so that the model continues learning smoothly instead of being disrupted. We analyze this problem using optimization theory and feature learning, and we test design choices such as how to initialize new layers, how to transfer hyperparameters, how to schedule the learning rate, and when to expand the model. Our proposed zero/one-layer progressive training method gives a strong tradeoff between training cost and final loss. On GPT-2, it can save about 80% of compute, or achieve about a 5× speedup, while reaching a loss comparable to training a full 60-layer, 7B-parameter model from scratch. Experiments on LLAMA3 and DeepSeekV3 model families further show 3–5× better compute efficiency, with larger gains at larger scales.