Towards Understanding the Dynamics of Low-Rank Adaptation

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning technique, and previous works have studied the update dynamics of LoRA, showing that updating via the low-rank matrix $\mathbf{A}$ can be viewed as a process within the compressed subspace spanned by $\mathbf{A}^{\top} \mathbf{A}$ of the gradient $\nabla f\left(\mathbf{W} \right)$. However, few works analyze how the properties of the low-rank matrices affect the performance of LoRA, since existing methods heuristically initialize the low-rank matrices as Gaussian matrices. In this paper, we provide a theoretical understanding of the update dynamics of LoRA. We reveal that the update dynamics can be viewed as a process within the subspace spanned by $\mathbf{A}^{\top} (\mathbf{A} \mathbf{A}^{\top})^{\dagger} \mathbf{A}$, and prove that when the gradient $\nabla f\left(\mathbf{W} \right)$ is unavailable, if $\mathbf{A}$ is an Equiangular Tight Frame (ETF), $\mathbf{A}^{\top} \mathbf{A}$ and $\mathbf{A}^{\top} (\mathbf{A} \mathbf{A}^{\top})^{\dagger} \mathbf{A}$ can preserve the maximum information from the gradient $\nabla f\left(\mathbf{W} \right)$. Thus, initializing $\mathbf{A}$ as an ETF is the optimal solution for low-rank adaptation when the gradient $\nabla f\left(\mathbf{W} \right)$ is unavailable. Furthermore, we establish the convergence of Low-Rank Adaptation with a rate of $\mathcal{O}\left(\frac{1}{T}\right)$ when $\mathbf{A}$ is an ETF. Extensive experiments show that initializing the low-rank matrices as ETFs significantly outperforms the commonly used Gaussian initialization for existing primary LoRA variants.

Fine-tuning large language models like GPT requires huge amounts of computing power, making it expensive and environmentally costly. Low-Rank Adaptation (LoRA) offers a cheaper alternative, but it usually starts with random numbers—like guessing a lock combination without any clues—which can be inefficient. We discovered that if we start the fine-tuning process with a special mathematical structure called an Equiangular Tight Frame (ETF), it acts like a master key. Our theory shows that this setup preserves the most important information from the unavailable full gradients, ensuring that the model learns faster and performs better. We tested this on various tasks, including question answering and code generation, across multiple LoRA variants. The results were clear: simply replacing the random start with our ETF structure consistently improved accuracy and accelerated convergence, sometimes by a large margin. This simple modification makes high-quality AI fine-tuning more efficient, reduces computational waste, and lowers the barrier for researchers with limited hardware resources.