Don't Forget Why You Started: Tackling Dual Forgetting in Vision-Language Continual Learning

Continual learning of Vision-Language Model (VLM) aims to empower models with new expertise without compromising zero-shot capabilities. However, this pursuit faces a critical “dual-forgetting” challenge: catastrophic forgetting of newly acquired classes (Incremental Knowledge Forgetting, IKF) and erosion of foundational zero-shot capabilities (Pre-trained Knowledge Forgetting, PKF). Existing evaluations often ignore PKF or assess it via confounded protocols where positive transfer on semantically similar domains creates an illusion of retention, masking severe foundational degradation. To address this, we propose the Dual-Forgetting-Aware Class-Incremental Learning (DFA-CIL) framework and the Similarity-Calibrated Retention (SCR) metric. Unlike standard averaging, SCR uses the frozen pre-trained feature space to inversely weight performance by semantic similarity, mitigating confounding gains to stress-test foundational stability. Building on this, we propose DFA-MoE, a functionally heterogeneous Parameter-Efficient Fine-Tuning (PEFT) method. DFA-MoE decouples optimization objectives by assigning a momentum-enhanced contrastive expert for feature alignment, and separate plasticity experts combining classification with auxiliary contrastive learning to adapt to new tasks while retaining historical knowledge. Extensive experiments show that our framework reveals the hidden fragility of existing methods and achieves a state-of-the-art balance in preserving both incremental and pre-trained knowledge. Our code is available at https://github.com/RL-MIND/DFA-MoE.

Vision-language models have strong multimodal understanding and zero-shot capabilities, but these abilities can degrade when they continually learn new categories. We propose a similarity-aware evaluation framework and metric to measure two types of forgetting: forgetting of previously learned classes and forgetting of pre-trained knowledge. By accounting for semantic similarity, our metric reduces misleading gains on similar domains and provides a more reliable measurement of pre-trained knowledge retention. We further introduce DFA-MoE, a parameter-efficient Mixture-of-Experts method that learns new categories while preserving zero-shot capabilities. This helps vision-language models better retain their pre-trained knowledge during continual learning.