Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning

LoRA adapts large language models (LLMs) by restricting updates to low-rank subspaces of pre-trained weights. While this substantially reduces training cost, the effectiveness of adaptation critically depends on which subspace is chosen at initialization: a poor initialization that allocates capacity to task-irrelevant directions can severely hinder downstream performance. Existing initialization strategies primarily rely on the intrinsic properties of pre-trained weights, implicitly assuming that weight geometry alone reflects task relevance. However, such criteria overlook how the model interacts with the downstream data distribution. In this work, we formulate LoRA initialization as identifying the degree of impact of directions in parameter space under the target data distribution. We argue that data-aware sensitivity, rather than weight-only magnitude, should govern the choice of adaptation subspaces. Building on this perspective, we propose a Fisher-guided framework that leverages curvature information induced by downstream data to characterize how parameter perturbations influence model predictions. This perspective yields a principled, task-dependent criterion for selecting LoRA directions that better align adaptation with the target objective. Empirical results across diverse tasks and modalities demonstrate that data-aware initialization consistently and significantly improves downstream performance over existing approaches.

Recent LoRA initialization methods mainly rely on the spectral structure of pre-trained weights, assuming that singular directions alone determine effective adaptation. However, these weight-only criteria ignore how the downstream data distribution interacts with the model, often allocating LoRA capacity to directions that are irrelevant for the target task. To address this limitation, we investigated whether data-aware parameter sensitivity could provide a more principled criterion for LoRA initialization. We propose FILet, a Fisher-guided initialization framework that identifies adaptation directions using Fisher Energy, a curvature-based measure derived from downstream data. Instead of selecting directions solely from singular values, FILet estimates Kronecker-factored Fisher statistics from minibatches, ranks candidate directions according to their data sensitivity, and initializes LoRA modules within low-Fisher-energy subspaces. This design enables efficient and scalable initialization without requiring expensive full Fisher computation or exact SVD decomposition. Our research matters because it shows that downstream adaptation quality depends not only on pre-trained weight geometry, but also on task-specific sensitivity induced by data. Across reasoning, generation, and image classification benchmarks, FILet consistently improves LoRA fine-tuning performance, demonstrating a practical and principled direction for data-aware PEFT initialization.