Learnability-Driven Knowledge Assimilation for Class-Incremental Semantic Segmentation

Class-incremental semantic segmentation learns new classes while retaining old ones without access to past data. Although existing methods alleviate catastrophic forgetting on old classes, new-class performance remains limited. We identify that the key bottleneck arises from low-margin regions, where the logit of the ground-truth class is close to that of the most competitive non-ground-truth class. Our theoretical analysis suggests that optimization in these regions is characterized by high second-order margin sensitivity and a small stability radius, making learning prone to class confusion. Based on the above analysis, we propose Learnability-Driven Knowledge Assimilation (LDKA), which targets low-margin learning via three complementary optimization strategies: (i) Progressive Margin Learning continuously reallocates pixel-wise optimization budget in a threshold-free manner, shifting emphasis from high-margin to low-margin regions; (ii) Smooth Knowledge Distillation applies second-order sensitivity damping along the margin direction and perturbation stabilization to suppress high-frequency updates and increase the stability radius; (iii) Misclassification-Aware Decoupling measures inter-class confusion with a competition matrix and decouples highly competitive class representations. Experiments show that LDKA improves mIoU on new classes while preserving performance on old classes across 9 incremental protocols.

AI systems for image understanding often need to learn new object categories over time. However, when a segmentation model learns new categories, it can forget older ones or confuse visually similar classes. This problem is especially important for semantic segmentation, where the model must assign a category label to every pixel in an image. This paper studies why this problem happens in class-incremental semantic segmentation. We find that many errors occur in uncertain image regions where the model gives similar scores to two competing classes. These low-margin regions are fragile: small model updates can easily change their predictions. To address this issue, we propose Learnability-Driven Knowledge Assimilation. The method gradually shifts training attention toward uncertain regions, stabilizes feature representations during knowledge transfer, and separates classes that are often confused. Experiments on standard segmentation benchmarks show that our method improves learning of new classes while helping preserve knowledge of old classes.