JANUS-LORA: A Balanced Low-Rank Adaptation for Continual Learning

Low-Rank Adaptation (LoRA) has emerged as a promising paradigm for Continual Learning. It independently updates its low-rank factors ($A$ and $B$), creating a composite update to the full weight matrix through their interaction. To prevent catastrophic forgetting, this update should remain orthogonal to the task-specific subspace that contains previously learned knowledge. However, we identify that this composite update systematically violates this orthogonality, reintroducing interference and undermining stability. Furthermore, naively enforcing this orthogonality compromises plasticity, disrupting the delicate stability-plasticity trade-off. To resolve these issues, we propose Janus-LoRA, a framework that restores this balance through two novel components. Specifically, we first introduce Gradient Rectification, a closed-form solution that mathematically decouples LoRA's factor updates, enforcing orthogonality against the historical knowledge subspace identified by an efficient Online Estimation. Next, to enhance plasticity, we introduce a Decoupled Margin Loss that promotes feature-level separation by pushing new feature representations away from old ones, thus creating distinct, low-interference regions for new learning. Comprehensive experiments on challenging benchmarks demonstrate that by harmonizing parameter-level orthogonality with feature-level separation, Janus-LoRA achieves a superior balance and establishes new state-of-the-art performance.

Modern AI models often need to learn new tasks over time. However, when they are updated for a new task, they may lose skills learned from earlier tasks, especially when old training examples cannot be stored. We propose Janus-LoRA, a lightweight method that helps an already-trained model keep old knowledge while learning new tasks. The method only updates small add-on parts of the model. It guides these updates so they interfere less with what the model learned before, while also helping the model keep new and old categories better separated. Our experiments show that Janus-LoRA performs better than existing methods on several continual learning benchmarks. This suggests a practical way to build AI systems that can keep learning over time without needing to save past user or training data.