MedCRP-CL: Continual Medical Image Segmentation via Bayesian Nonparametric Semantic Modality Discovery

Medical image segmentation faces a fundamental challenge in continual learning: data arrives sequentially from heterogeneous sources, yet effective continual learning requires discovering which tasks share sufficient structure to benefit from joint learning. Existing methods either apply uniform constraints across all tasks, causing catastrophic forgetting when tasks conflict, or require predefined task groupings that cannot anticipate future task diversity. We introduce MedCRP-CL, a framework that performs online task structure discovery and structure-aware continual learning. Leveraging the Chinese Restaurant Process (CRP), our method dynamically infers task groupings from clinical text prompts as tasks arrive, without requiring predefined cluster counts or access to future tasks. We term these discovered groupings semantic modalities, as they capture finer-grained structure than physical imaging modalities by integrating anatomical region and pathological context. Guided by this discovered structure, we maintain semantic modality-specific LoRA adapters regularized by intra-modality EWC, ensuring parameter isolation across dissimilar task groups while facilitating knowledge transfer within similar ones. The framework is also replay-free, storing only aggregate statistics rather than raw patient data. Experiments on 16 medical segmentation tasks across four imaging modalities demonstrate that MedCRP-CL achieves 73.3\% Dice score with only 4.1\% forgetting, outperforming the best baseline by 8.0\% while requiring 6$\times$ fewer parameters. Code is available at https://github.com/zygao930/MedCRP-CL.

AI systems that analyze medical images, such as X-rays or ultrasounds, often need to learn new tasks over time as hospitals adopt new imaging equipment or encounter new conditions. However, when these systems learn something new, they tend to forget what they previously learned. Existing approaches either treat all tasks the same, leading to interference between unrelated tasks, or require doctors to manually specify how tasks should be grouped, which is impractical as new data arrives unpredictably. We developed MedCRP-CL, a system that automatically discovers which medical imaging tasks should share knowledge and which should be kept separate. It does this by reading the clinical descriptions that accompany medical images and using a statistical method to discover natural groupings. For example, it recognizes that heart ultrasounds and breast ultrasounds, despite both being ultrasounds, require very different analysis strategies. Each discovered group gets its own specialized parameters, preventing interference while still allowing related tasks to help each other. Tested on 16 medical imaging tasks, our approach outperforms existing methods while using fewer parameters. It also requires no storage of patient data, making it suitable for privacy-sensitive clinical settings.