SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning

Progressive Learning (PL) reduces pre-training computational overhead by gradually increasing model scale. While prior work has extensively explored depth expansion, width expansion remains significantly understudied, with the few existing methods limited to the early stages of training. However, expanding width during the mid-stage is essential for maximizing computational savings, yet it remains a formidable challenge due to severe training instabilities. Empirically, we show that naive initialization at this stage disrupts activation statistics, triggering loss spikes, while copy-based initialization introduces gradient symmetry that hinders feature diversity. To address these issues, we propose **SPARKLING** (balancing **S**ignal **P**reservation **A**nd symmet**R**y brea**K**ing for width-progressive **L**earn**ING**), a novel framework for mid-stage width expansion. Our method achieves signal preservation via RMS-scale consistency, stabilizing activation statistics during expansion. Symmetry breaking is ensured through asymmetric optimizer state reset and asymmetric learning rate re-warmup. Extensive experiments on dense and Mixture-of-Experts (MoE) models demonstrate that, across multiple width axes and optimizer families, SPARKLING consistently outperforms training from scratch and reduces training cost by up to 35% under $2\times$ width expansion.

Training today's large language models consumes enormous amounts of compute and energy. A promising remedy, called Progressive Learning, starts with a small model and gradually grows it while training, so that most of the work happens on a cheaper model. Prior research has mostly worked out how to add new layers (growing the network's "depth"), but enlarging each layer (growing its "width") partway through training has remained notoriously unstable: training loss suddenly spikes, the model struggles to converge, or the freshly added capacity is simply wasted. We trace this instability to two hidden failure modes. First, when new components are inserted, the typical signal strength inside the network is disturbed; we fix this by carefully rescaling the new components so the internal contribution of the network stays the same. Second, when extra units are created by duplicating existing ones, the copies are mathematically locked together and never specialize. We unlock them with two simple, asymmetric interventions --- resetting the optimizer's memory and re-warming the learning rate --- applied only to the new units. Our recipe, SPARKLING, matches or surpasses models trained from scratch while cutting training cost by up to 35% when doubling model width, making large language model training noticeably cheaper and more accessible.