Mixing Expertise with Confidence: A Mixture of Experts Framework for Robust Multi-Modal Continual Learning

The Mixture of Experts (MoE) framework is widely used in continual learning to mitigate catastrophic forgetting. MoEs typically combine a small inter-task shared parameter space with largely independent expert parameters. However, as the number of tasks increases, the shared space becomes a bottleneck, reintroducing forgetting, while fully independent experts require explicit task ID predictors (e.g., routers), adding complexity. In this work, we eliminate the inter-task shared parameter space and the need for a task ID predictor by enabling expert communication and allowing knowledge to be shared dynamically, akin to human collaboration. We bridge the inter-expert knowledge sharing by leveraging the open-set learning capabilities of a multimodal foundation model (e.g., CLIP), thereby providing “expert priors” that bolster each expert’s task-specific representations. Guided by these priors, experts learn calibrated inter-task posteriors. Additionally, multivariate Gaussians over the learned posteriors promote complementary specialization among experts. We propose new evaluation benchmarks that simulate realistic continual learning scenarios, and our prior-conditioned strategy consistently outperforms existing methods across diverse settings without relying on reference datasets or replay memory.

Artificial intelligence (A.I.) systems, like humans, need to learn new tasks over time. However, when learning something new, they often forget previously learned knowledge, a problem known as catastrophic forgetting. One promising solution is the Mixture of Experts (MoE) approach, where several smaller A.I. models (“experts”) work together, and only a few are activated for a given task. Yet, because the same experts continue learning multiple tasks over time, they can still forget earlier knowledge and become overly confident when encountering data that resembles previous tasks, relying too much on shared patterns instead of the most relevant features of the current data. To address this, we propose an expanding expert framework that continuously adds new experts over time. This avoids retraining old experts or requiring a task identifier, helping prevent forgetting. Yet, experts who learn independently can become overconfident due to limited communication. We introduce confidence-based communication between experts to help them recognize familiar patterns and learn more cautiously. During prediction, experts can rely on their own predictions or defer to more confident experts. We also use a weighting mechanism to identify the best expert for each image. Across several continual learning benchmarks, our approach improves robustness without storing old data, using extra datasets, or requiring a separate expert selector.