On the Convergence Rate of LoRA Gradient Descent

The low-rank adaptation (LoRA) algorithm for fine-tuning large models has grown popular in recent years due to its remarkable performance and low computational requirements. LoRA trains two "adapter" matrices that form a low-rank representation of the model parameters, thereby massively reducing the number of parameters that need to be updated at every step. Although LoRA is simple, its convergence is poorly understood due to the lack of Lipschitz smoothness, a key condition for classic convergence analyses. As a result, current theoretical results only consider asymptotic behavior or assume strong boundedness conditions which artificially enforce Lipschitz smoothness. In this work, we provide for the first time a non-asymptotic convergence analysis of the *original LoRA gradient descent* algorithm, which reflects widespread practice, without such assumptions. Our work relies on three key steps: i) reformulating the problem in terms of the outer product of the stacked adapter matrices, ii) a modified descent lemma for the "Lipschitz-like" reparametrized function, and iii) controlling the step size. With this approach, we prove that LoRA gradient descent converges to a stationary point at rate $O(\frac{1}{\log T})$, where $T$ is the number of iterations.

Large machine learning models are frequently updated using an algorithm called Low-Rank Adaptation (LoRA), which reduces the number of parameters requiring modification to lower computational requirements. Despite its popularity, the mathematical characterization of LoRA remains undeveloped. Traditional mathematical tools for proving that an algorithm successfully reaches a solution require a continuous bounding property known as Lipschitz smoothness. LoRA lacks this property, leading prior theoretical studies to rely on strong assumptions to prove convergence. This work provides a direct convergence analysis of the standard LoRA gradient descent algorithm without requiring these constraints. We achieve this by mathematically reformulating the optimization problem, developing a modified analytical framework to accommodate the lack of standard smoothness, and applying precise constraints on the algorithm's step size. Through this method, we mathematically prove that the algorithm converges to a stable state at a specific, quantifiable rate over a finite number of iterations.