CoPE: Continual Probe-guided Expansion for Large Vision-Language Models

Mixture of Experts architectures have recently advanced the scalability and adaptability of Large Language Models for continual multimodal learning. However, extending these models to accommodate sequential tasks remains challenging. As new tasks arrive, naive model expansion leads to rapid parameter growth, while modifying shared routing components often causes catastrophic forgetting, undermining previously learned knowledge. To address these issues, we propose CoPE, a continual learning framework for LLMs that requires no replay data of previous tasks and ensures both parameter efficiency and robust knowledge retention. Our approach introduces the Probe-Guided Knowledge Extension mechanism, which uses probe experts to dynamically determine when and where new experts should be added, enabling adaptive and minimal parameter expansion tailored to task complexity. To support inference without task labels, we further incorporate a Probabilistic Task Locator that dynamically matches inputs to the correct task-specific components. To handle the practical issue that task labels are unknown during inference, we leverage a VAE-based reconstruction strategy to identify the most suitable router by matching input distributions, allowing automatic and accurate expert allocation. This design mitigates routing conflicts and catastrophic forgetting, enabling robust continual learning without explicit task labels. Extensive experiments on the CoIN benchmark, covering eight diverse VQA tasks, demonstrate that CoPE delivers strong continual learning performance with a compact model size, significantly reducing forgetting and parameter overhead compared to prior methods. These results showcase the effectiveness and scalability of our approach for parameter-efficient continual learning in large language models. Our code will be open-sourced soon.

As artificial intelligence models learn to perform a continuous sequence of new tasks, they often struggle with "catastrophic forgetting"—losing previously acquired knowledge—or become inefficiently large by blindly adding too many new parameters. To solve this, we introduce COPE, a continual learning framework for Large Vision-Language Models that remembers past tasks without needing to store old data. COPE works by deploying small "probe experts" to test the waters, dynamically adding new capacity only when and where the model truly needs it to handle a new challenge. Additionally, it features an automatic task locator that identifies the correct set of skills to use for any given input, even when the task type is not explicitly provided. Ultimately, our approach allows AI models to efficiently learn new multimodal abilities over time while keeping their size compact and their memories intact.