Turning Back Without Forgetting: Selective Backward Refinement for Parameter-Efficient Continual Learning

While prompt-based parameter-efficient continual learning mitigates catastrophic forgetting by isolating task-specific prompts, this isolation also limits later tasks from improving earlier ones, leaving backward knowledge transfer underexplored. We address this limitation by proposing Selective bAckward refinement for positive Backward knowledge transfER (SABER), a replay-free framework that enables controlled backward transfer in prompt-based continual learning. SABER determines when backward refinement is beneficial using complementary task-correlation criteria based on prompt-gradient geometry and loss-distribution similarity, and how to perform refinement safely by restricting updates to non-interfering directions in the prompt parameter space. Extensive experiments across multiple continual learning benchmarks, and diverse pretrained backbones, including T5-Large, LLaMA, and Qwen, demonstrate that SABER consistently achieves positive backward transfer while maintaining strong overall average performance.

Modern AI models often need to learn many tasks over time, but learning a new task can hurt performance on earlier tasks. A common solution is to give each task its own small prompt while keeping the main model fixed. This helps prevent forgetting, but it also means that earlier task prompts are never improved using knowledge from later related tasks. In this work, we propose SABER, a method that decides when a later task can help an earlier one and then safely updates only the useful parts of the earlier prompt. SABER does this without storing or replaying old training data. Experiments across multiple continual learning benchmarks and language models show that SABER improves earlier tasks while maintaining strong overall performance.