Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge. Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge. From a knowledge-decomposition perspective, we observe that low-rank adaptations exhibit highly imbalanced singular value spectra: a few dominant components absorb most of the adaptation energy, thereby (i) more likely to disrupt previously acquired knowledge and (ii) making the update more vulnerable to interference from subsequent tasks. To enable explicit balance among components, we decouple the magnitude of the task update from its directional structure and formulate it as a constrained optimization problem on a restricted Stiefel manifold. We address this problem using a projected first-order method compatible with standard deep-learning optimizers used in vision-language models. Our method mitigates both backward and forward forgetting, consistently outperforming continual learning baselines. Source code is available in supplementary material.

Modern AI systems that understand both images and text often need to learn new tasks over time, but updating them can cause them to forget earlier abilities. A common lightweight update method changes only a small part of the model, yet we found that these changes are often uneven: a few internal directions carry most of the learning while many others are underused. This imbalance is like putting too much pressure on a few bridges between old and new knowledge, making them easier to damage and more likely to interfere with each other. We introduce EBLoRA, a new way to make these lightweight updates more balanced. EBLoRA separates how big an update is from where it points, spreads learning more evenly across internal directions, and avoids directions that were important for previous tasks. It can be trained with standard deep-learning tools, so it is practical to use with large vision-language models. In experiments on several image-and-text continual learning benchmarks, EBLoRA better preserved earlier knowledge and improved transfer to future tasks compared with strong existing methods. This helps AI systems adapt to changing real-world data while remaining more reliable over time.