Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence

Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show—both theoretically and empirically—that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.

Large language models (LLMs) are typically adapted to specific tasks through a technique called Low-Rank Adaptation (LoRA), which fine-tunes only a small number of additional parameters rather than the full model. While efficient, LoRA has a subtle redundancy: many different sets of parameters can produce the exact same model update. We show that this redundancy has real consequences — some of these equivalent parameter sets make the optimization landscape much harder to navigate, slowing down training. To fix this, we introduce BaLoRA, a simple extension of LoRA that steers the parameters toward a "well-behaved" region during training. This correction is computationally cheap and easy to plug into existing workflows. In practice, BaLoRA trains faster and achieves better performance than standard LoRA across a variety of tasks.