Merge to Remember: Sharpness-Aware Isotropic Merging for Continual Learning

Continual learning with large pre-trained models offers significant potential for cross-task knowledge accumulation, but faces critical challenges such as catastrophic forgetting and parameter interference, especially when historical data is unavailable. Existing approaches typically rely on sequential fine-tuning or model merging strategies, yet often overlook the impact of loss landscape sharpness and dominant singular value directions, which leads to subspace misalignment and severe knowledge forgetting. In this paper, we propose the Sharpness-Aware Isotropic Merging (SAIM) framework, which introduces targeted optimizations in both the fine-tuning and merging stages to address these issues. Specifically, SAIM consists of two synergistic modules: (1) a Sharpness-Aware Block Coordinate Descent (SA-BCD) optimizer that guides the model toward flatter minima and selectively updates the most task-sensitive parameters, thereby mitigating parameter interference and enhancing robustness; (2) an adaptive isotropic merging algorithm that dynamically balances the singular value spectrum across tasks, effectively preventing the model from overemphasizing any single task direction, maintaining balanced knowledge representation, and improving subspace alignment. Extensive experiments on vision and language benchmarks demonstrate that SAIM achieves 5-10\% higher accuracy than existing methods and maintains robust performance as the number of tasks increases. Our code are available at \url{https://github.com/Yangqun123456/SAIM}.

Modern AI systems are often asked to learn new tasks one after another, such as new image categories or new language skills. A major problem is that they often forget what they learned earlier when trained on something new. This is especially hard when old training data cannot be stored or reused because of privacy, cost, or storage limits. Existing methods usually either keep fine-tuning the same model or combine separate task models afterward, but both approaches can let one task dominate and weaken older abilities. Our paper introduces SAIM, a method designed to help models keep what they already know while still learning something new. First, it trains the model to settle into more stable settings, so small later changes are less likely to damage earlier knowledge. Then it carefully rebalances the updates from different tasks so that no single task takes over the shared model. Across a wide range of image and language benchmarks, SAIM improves accuracy by 5 to 10 percentage points over strong baselines and stays reliable as the number of tasks grows. This could help build AI systems that learn continuously without needing to keep old data, which is important for privacy-sensitive and storage-limited applications.