RA-LoRA: Rank-Adaptive Low-Rank Adaptation via Subspace Interference Measurement for Continual Fine-Tuning of Foundation Models

Sequential fine-tuning of foundation models with LoRA (Hu et al., 2022) suffers from catastrophic forgetting: new adapter updates overwrite previously acquired skills. We introduce RA-LoRA, which before each new task adaptation measures the principal angle $\theta_1$ between the incoming gradient subspace and the accumulated adapter subspace. We prove that forgetting is bounded by $C \sin^2 \theta_1$, establishing the principal angle as a theoretically principled proxy for interference. RA-LoRA uses this measure to make an online binary decision: when $\theta_1 \le \theta*$ (low interference), it projects gradients into the orthogonal complement of $\mathcal{S}_{1:t-1}$ and shares parameters; when $\theta_1 > \theta^*$ (high interference), it appends a small fresh rank-$\Delta r$ module. Unlike O-LoRA (Wang et al., 2023), which always enforces strict orthogonality and grows rank linearly, RA-LoRA grows rank sub-linearly—only when genuinely needed—while allowing partial subspace sharing for beneficial forward transfer. On a 5-task sequential instruction-tuning benchmark with T5-large (Raffel et al., 2020), RA-LoRA achieves 75.5\% average accuracy, outperforming O-LoRA (72.4\%), EWC+LoRA (55.7\%), and sequential LoRA (41.9\%) while using 55\% fewer parameters than O-LoRA.