Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating

Low-Rank Adaptation (LoRA) achieves parameter-efficient fine-tuning by constraining model updates to a low-rank subspace and has been widely used in practice. However, LoRA typically employs a shared low-rank update across tokens, which limits its ability to fully exploit the adaptation subspace for tokens from different sequences. To address this issue, we propose an adaptive utilization of Low-Rank Adaptation (U-LoRA), which employs conditioned gating to explicitly learn effective token-level utilization of the limited low-rank adaptation subspace. Specifically, U-LoRA generates utilization coefficients along low-rank directions for each token and jointly coordinates and constrains them using sequence-level contextual information, thereby inducing more consistent adaptive patterns within a sentence. To further enhance training stability, we introduce a bias-corrected exponential moving average (EMA) historical prior that calibrates utilization signals across optimization steps, suppressing noise caused by batch-to-batch fluctuations. The effectiveness of our method arises from a better utilization of the existing low-rank subspace via input-conditioned strategies, rather than from expanding the subspace. Experiments on mathematical reasoning and natural language understanding benchmarks demonstrate that U-LoRA achieves competitive performance under comparable parameter budgets when with strong LoRA baselines and recent variants.

Fine-tuning large language models for new tasks is expensive because it normally requires updating billions of parameters. LoRA offers a clever shortcut: instead of touching the whole model, it learns only a small, low‑cost “patch.” However, standard LoRA applies the same patch uniformly to every word in a sentence, ignoring that some words matter more than others for a given task. What if different tokens could use different amounts of LoRA adaptation rather than a uniform update? We introduce U‑LoRA, instead of a uniform update, our method dynamically adjusts how the model adapts to each specific word based on the overall context of the sentence. We also incorporated a smoothing technique during the training phase to keep the model's learning process stable and reduce random noise. Essentially, we teach the model to make much smarter use of the limited adjustment space it is already given, rather than just giving it more space. Our experiments show that this targeted approach significantly improves the model's ability to solve math problems and understand natural language.