Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts

Continual learning, especially class-incremental learning (CIL), on the basis of a pre-trained model (PTM) has garnered substantial research interest in recent years. However, how to effectively learn both discriminative and comprehensive feature representations while maintaining stability and plasticity over very long task sequences remains an open problem. We propose $\mathbf{CaRE}$, a scalable $\mathbf{C}$ontinual Le$\mathbf{a}$rner with efficient Bi-Level $\mathbf{R}$outing Mixture-of-$\mathbf{E}$xperts (BR-MoE). The core idea of BR-MoE is a bi-level routing mechanism: a router selection stage that dynamically activates relevant task-specific routers, followed by an expert routing phase that dynamically activates and aggregates experts, aiming to inject discriminative and comprehensive representations into every intermediate network layer. On the other hand, we introduce a challenging dataset, OmniBenchmark-1K, for CIL performance evaluation on very long task sequences with hundreds of tasks. Extensive experiments show that CaRE demonstrates leading performance across a variety of datasets and task settings, including commonly used CIL datasets with classical CIL settings (e.g., 5-20 tasks). To the best of our knowledge, CaRE is the first continual learner that scales to very long task sequences (ranging from 100 to over 300 non-overlapping tasks), while outperforming all baselines by a large margin on such task sequences. We hope that this work will inspire further research into continual learning over extremely long task sequences. Code and dataset are publicly released at https://github.com/LMMMEng/CaRE.

When AI models learn new tasks, they often forget what they learned before, a problem called "catastrophic forgetting". To address this, Continual Learning (CL) has become a promising solution. However, existing methods work well for a small number of tasks (e.g., 20 tasks), but it is unclear whether they remain effective when faced with hundreds of consecutive tasks. Additionally, there was no suitable benchmark to evaluate a model's ability to learn over such long task sequences. We propose CaRE, a strong CL method that can dynamically retrieve and combine relevant knowledge from past tasks when learning new ones, mimicking how humans learn. We also created OmniBenchmark-1K, a challenging dataset with 1,000 diverse classes designed specifically for evaluating very long-sequence CL. CaRE is the first CL system to effectively handle over 300 non-overlapping tasks, outperforming all existing methods by a large margin. Our code and dataset are publicly available, enabling further research into more practical AI systems that can learn continuously in real-world environments.