LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis

LoRA has become a widely adopted method for PEFT, and its initialization methods have attracted increasing attention. However, existing methods have notable limitations: many methods do not incorporate target-domain data, while gradient-based methods exploit data only at a shallow level by relying on one-step gradient decomposition. In this paper, we establish a theoretical framework for data-aware LoRA initialization. Starting from minimizing the expectation of the parameter discrepancy between the fine-tuned and target models, we derive an optimization problem with two components: a bias term, which is related to the parameter distance between the fine-tuned and target models, and is approximated using a Fisher–gradient formulation to preserve anisotropy; and a variance term, which accounts for the uncertainty introduced by sampling stochasticity through the Fisher information. Solving this problem yields an optimal initialization strategy for LoRA, based on which we develop an efficient algorithm, LoRA-DA. Empirical results across multiple benchmarks demonstrate that LoRA-DA consistently improves final accuracy over existing initialization methods. Additional studies show faster, more stable convergence, robustness across ranks, and only a small initialization overhead for LoRA-DA. The source code is available at https://github.com/zqy0126/LoRA-DA.

Adapting large language models to new tasks is usually expensive because changing all of their parameters requires large amounts of memory and computation. A popular workaround, called low-rank adaptation, changes only a small set of added parameters, but these new parameters are usually started almost randomly, which can make training slower or less effective. Our work asks whether the model can be given a better starting point before fine-tuning begins. We propose LoRA-DA, a method that looks at a small set of examples from the target task and uses them to identify useful directions for the model to adapt. It also accounts for which directions in the model are more reliable and which are more uncertain, rather than relying only on a first gradient step. Across language understanding and mathematical reasoning benchmarks, this initialization improves final accuracy, makes training more stable, and adds only a small extra cost. This can make efficient fine-tuning of large models more reliable for researchers and practitioners with limited computing resources.