Curriculum-Guided Layer Scaling for Language Model Pretraining

As the cost of pretraining large language models grows, there is continued interest in strategies to improve learning efficiency during this core training stage. Motivated by cognitive development, where humans gradually build knowledge as their brains mature, we propose Curriculum-Guided Layer Scaling (CGLS), a framework for compute-efficient pretraining that synchronizes increasing data difficulty with model growth through progressive layer stacking (i.e. gradually adding layers during training). At the 100M parameter scale, using a curriculum transitioning from synthetic short stories to general web data, CGLS outperforms baseline methods on the question-answering benchmarks PIQA and ARC. Pretraining at the 1.2B scale, we stratify the DataComp-LM corpus with a DistilBERT-based classifier and progress from general text to highly technical or specialized content. Our results show that progressively increasing model depth alongside sample difficulty leads to better generalization and zero-shot performance on various downstream benchmarks. Altogether, our findings demonstrate that CGLS unlocks the potential of progressive stacking, offering a simple yet effective strategy for improving generalization on knowledge-intensive and reasoning tasks.

We propose Curriculum-Guided Layer Scaling (CGLS), a method for training language models more efficiently by growing the model and increasing the difficulty of the data at the same time. Instead of training the full model on all data from the start, CGLS begins with a smaller model on simpler text, gradually adds layers, and shifts toward harder or more specialized data. Across GPT-2-Small and LLaMA-3.2-1B scale experiments, CGLS improves performance on several reasoning and knowledge-focused benchmarks compared with standard training, curriculum-only training, and layer-scaling-only baselines. The results suggest that progressive model growth is most useful when paired with a matching curriculum over data complexity.