Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning

Low-Rank Adaptation (LoRA) is a standard tool for parameter-efficient finetuning of large models. While it induces a small memory footprint, its training dynamics can be surprisingly complex as they depend on several hyperparameters such as initialization, adapter rank, and learning rate. In particular, it is unclear *how the optimal learning rate scales with adapter rank*, which forces practitioners to re-tune the learning rate whenever the rank is changed. In this paper, we introduce *Maximal-Update Adaptation* ($\mu$A), a theoretical framework that characterizes how the "optimal" learning rate should scale with model width and adapter rank to produce stable, non-vanishing feature updates under standard configurations. Our analysis leverages techniques from hyperparameter transfer and reveals that the optimal learning rate exhibits different scaling patterns depending on initialization and LoRA scaling factor. Specifically, we identify two regimes: one where the optimal learning rate remains roughly invariant across ranks, and another where it scales inversely with rank. We further identify a configuration that allows learning rate transfer from LoRA to full finetuning, drastically reducing the cost of learning rate tuning for full finetuning. Experiments across language, vision, vision–language, image generation, and reinforcement-learning tasks validate our scaling rules and show that learning rates tuned on LoRA transfer reliably to full finetuning.

Today's AI models are first trained on huge amounts of data, then "finetuned" — adjusted to do a specific job. Updating every part of such a large model is expensive, so practitioners often use LoRA, a lightweight method that trains only a small add-on rather than the whole model. But LoRA has several settings that interact in complicated ways. The most important of these is the learning rate, which sets how aggressively the model adjusts as it learns: its best value shifts whenever you resize the add-on, forcing slow trial-and-error retuning. We developed a mathematical framework that predicts how the learning rate should scale with the add-on's size, keeping learning in a stable sweet spot — neither overshooting nor stalling. We also discovered that, in one common setup, the best learning rate for LoRA coincides with the one needed to fully update the model. Researchers can therefore tune this setting cheaply with LoRA and reuse it for expensive full finetuning, skipping costly searches. This saves time, money, and energy, and lowers the barrier for those with limited computing power. We confirmed these rules across language, vision, image-generation, and reinforcement-learning tasks.