Provable Forgetting Bounds Drive Capacity Savings: Spectral Thresholding in Continual LoRA

Orthogonal-subspace LoRA methods mitigate forgetting in continual learning of foundation models by assigning each task a new adapter subspace, but existing approaches typically allocate a fixed, hand-picked rank to every task and layer. This ignores a key source of heterogeneity: the amount of rank needed to avoid interference depends on the residual spectrum of the task's layerwise activations. We introduce **DyRA**, a theory-guided dynamic rank allocation rule that selects adapter bases by retaining residual singular directions whose singular value exceeds a global threshold $\tau$. Our method is motivated by a per-layer interference bound showing that, for orthogonal LoRA, future-task interference on a past task is controlled by the spectral tail left outside the past-task adapter basis. Thus, **DyRA** replaces a fixed-rank design choice with a spectral-tail control principle. On continual instruction tuning of the LLaMA-2 7B foundation model with TRACE, **DyRA** yields a favorable performance–capacity trade-off, with the clearest gains at low rank ($+4.7$ AP over fixed $r=4$) and matches fixed $r=8$ with about $31\%$ fewer adapter parameters. **DyRA** requires only one residual-spectrum computation per task and layer, and can be added to existing orthogonal-subspace LoRA methods.