The GL(r) Gauge Symmetry of LoRA: Principal Bundle Structure, Loss Landscape Geometry, and Implications for Adapter Merging

Low-Rank Adaptation (LoRA) parametrizes weight updates as $\Delta W = BA$ and is widely used for efficient fine-tuning of large language models[cite: 3]. We show that this factorization endows the parameter space with a continuous gauge symmetry: the group $GL(r, \mathbb{R})$ acts freely on $(A, B)$ pairs while leaving $\Delta W$ invariant[cite: 3]. We prove that this structure makes $(E_r, \pi, M_r)$ a smooth principal $GL(r)$-bundle over the manifold of rank-$r$ weight updates, implying exactly $r^2$ flat directions in the loss landscape at every critical point[cite: 3]. We derive a closed-form expression for the gauge-induced error in linear interpolation of LoRA adapters, showing that widely-used merging methods suffer from gauge ambiguity rather than fundamental task incompatibility[cite: 3]. We propose gauge-aligned merging (Algorithm 1) based on a canonical SVD gauge section, prove its consistency, and validate it empirically on multi-task LoRA merging with LLaMA-3-8B[cite: 3]. Gauge-aligned merging matches full-rank task arithmetic at a fraction of the cost and significantly outperforms naive parameter averaging[cite: 3]. Our framework reveals that LoRA strictly enlarges the symmetry group of fine-tuning and that existing permutation-based alignment methods handle only the measure-zero subgroup $S_r \subset GL(r)$[cite: 3].