CSPLoRA: Confidence-Guided Structure Planning for Low-Rank Adaptation

Low-Rank Adaptation (LoRA) has become the de facto paradigm for parameter-efficient fine-tuning, with its effectiveness critically influenced by rank allocation across modules. However, existing approaches face a fundamental dilemma: uniform allocation ignores module heterogeneity, while adaptive methods introduce expensive training overhead or lack reusability across configurations. We propose CSPLoRA (Confidence-guided Structural Planning for LoRA), a decoupled framework that reweights probe samples by prediction uncertainty to obtain more discriminative module importance estimates. The key insight is that hard samples---those the model struggles with---provide more informative gradient signals for identifying critical modules than easy samples. For a fixed task-model pair, the resulting structural priors can be reused across compatible rank budgets and LoRA backends, supporting a practical "probe once, deploy everywhere" workflow. Experiments on GLUE, commonsense reasoning, and arithmetic tasks show that CSPLoRA improves over uniform LoRA on average (+1.25 points on LLaMA-2-7B commonsense reasoning) while maintaining comparable parameters, with the planned rank structure reusable across compatible LoRA variants.

Adapting a large language model to a new task can be expensive, even when only a small number of trainable parameters are used. A popular method called LoRA makes this cheaper by adding small trainable components to many parts of the model, but it usually gives every part the same size. This is not ideal because different parts of a model may matter differently for a particular task. We propose CSPLoRA, a method that first runs a short probing stage before normal fine-tuning. During this probe, the method pays more attention to examples that the model finds difficult, because these examples reveal which parts of the model are more important for the new task. CSPLoRA then assigns more trainable capacity to important parts and less to less important parts, while keeping the overall parameter budget comparable to standard LoRA. This makes fine-tuning more targeted without changing the underlying model architecture. Across language understanding, commonsense reasoning, and math reasoning tasks, CSPLoRA improves over uniform LoRA baselines on average. The planned structure can also be reused across compatible LoRA variants and rank budgets, reducing the cost of repeated experimentation.