Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods.

Modern AI models often need to continually learn new tasks over time, but a major challenge is that learning a new task can make the model forget what it learned before. Recently, Low-Rank Adaptation, or LoRA, is an efficient way to adapt large models by updating only a small number of parameters, and it has received increasing attention in continual learning. Existing LoRA-based continual learning methods usually reduce conflicts between tasks by separating their update directions, often choosing directions that previous tasks seem not to use. However, they always (i) ignore directions that are useful across multiple tasks, which limits knowledge transfer, and (ii) fail to find truly useful task-specific directions, because directions that look unused by old tasks can also be weakly useful for a new task when tasks are related. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). LoDA separates the update space into two parts: a general part that captures knowledge shared by old and new tasks, and a task-specific part that is useful for the new task while having little effect on old tasks. It then learns robust updates in these two parts via a Gradient-Aligned Optimization (GAO) approach. After each task, LoDA further recalibrates the shared update with a direct closed-form adjustment before merging it into the model, so that the shared update better balances old and new tasks. Experiments show that LoDA performs better than existing continual learning methods.