ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning

As large language models (LLMs) continue to scale in size, the computational overhead has become a major bottleneck for task-specific fine-tuning. While low-rank adaptation (LoRA) effectively curtails this cost by confining the weight updates to a low-dimensional subspace, such a restriction can hinder effectiveness and slow convergence. This contribution deals with these limitations by accumulating progressively a high-rank weight update from consecutive low-rank increments. Specifically, the per update optimal low-rank matrix is identified to minimize the loss function and closely approximate full fine-tuning. To endow efficient and seamless optimization without restarting, this optimal choice is formed by appropriately scaling the columns of the original low-rank matrix. Rigorous performance guarantees reveal that the optimal scaling can be found analytically. Extensive numerical tests with popular LLMs scaling up to 12 billion parameters demonstrate a consistent performance gain and fast convergence relative to state-of-the-art LoRA variants on diverse tasks including natural language understanding, commonsense reasoning, and mathematical problem solving.

Large language models are often customized for new tasks using lightweight methods such as LoRA, which greatly reduce the cost of fine-tuning by updating only a small number of parameters. However, this efficiency can come at the expense of accuracy and slower learning. This paper introduces ScaLoRA, a method that progressively combines many small low-rank updates into a richer high-rank update while still remaining computationally efficient. ScaLoRA also avoids costly optimization restarts by analytically rescaling existing updates during training. Experiments on language understanding, commonsense reasoning, and mathematical problem solving show that ScaLoRA consistently improves both performance and convergence speed over existing LoRA-based methods, while remaining scalable to modern billion-parameter language models.