$\mu$pscaling small models: Principled warm starts and hyperparameter transfer

Modern large-scale neural networks are often trained and released in multiple sizes to accommodate diverse inference budgets. To improve efficiency, recent work has explored *model upscaling*: initializing larger models from trained smaller ones to accelerate convergence. However, this method can be sensitive to hyperparameters that need to be tuned at the target upscaled model size, which is prohibitively costly to do directly. It remains unclear whether tuning hyperparameters on smaller models and extrapolating via scaling laws is sound in this setting. We address this with principled approaches to width-based upscaling and efficient hyperparameter tuning in this setting. Motivated by $\mu$P and any-dimensional architectures, we introduce a general upscaling method that, like Net2Net, copies and perturbs weights, but uses theoretically grounded, width-dependent scalings for the perturbation noise and optimizer hyperparameters. First, we prove that under zero perturbation, the upscaled model is functionally equivalent to the base model throughout training. Second, we extend the $\mu$P theory to enable infinite-width limit analysis and establish hyperparameter transfer for upscaled models, greatly reducing the tuning cost. We empirically demonstrate that this method is effective on realistic datasets and architectures.

Training large AI models from scratch is expensive. A promising shortcut called "model upscaling" initializes a large model by copying weights from a smaller, already-trained one — much like promoting an employee who already knows the job, rather than hiring and training from scratch. The catch is that the fine-grained settings governing training success, such as the learning rate, must be carefully retuned for such large models — a process that can itself be very costly. Our work solves this problem using a mathematical theory that describes how the right training settings change as a neural network grows in size. We introduce an upscaling method that copies and gently perturbs the smaller model's weights using theoretically prescribed scalings. First, we prove that our upscaled model behaves exactly like the original if we copy without perturbing — giving a clean baseline interpretation of the algorithm. Second, we prove that the optimal settings found at a smaller scale transfer directly to the upscaled model, making tuning far cheaper. Experiments on realistic models and datasets confirm that the approach works in practice. Together, these results make upscaling both principled and cost-effective, potentially reducing the computational burden of deploying AI systems at scale.