Continual Segmentation under Joint Nonstationarity

Evolving data streams induce joint nonstationarity in continual semantic segmentation, where semantic classes, input distributions, and supervision availability change simultaneously over time. This setting reflects practical structured prediction systems, yet remains largely unexplored in prior continual learning work, which typically studies these factors in isolation. We formalize continual segmentation under coupled class, domain, and label shifts and investigate learning in heterogeneous dense prediction environments with limited annotations and abundant unlabeled data. To address instability and overfitting arising from few-shot supervision under distribution drift, we introduce gradient-adaptive stabilization, a parameter-wise regularization mechanism implemented via gradient-scaled stochastic perturbations that promotes a principled stability–plasticity tradeoff. We further leverage unlabeled data through semi-supervised learning and introduce prototype anchored supervision that validates pseudo-labels via joint confidence and prototype consistency. Together, these mechanisms enable learning under joint nonstationarity in continual segmentation. Extensive empirical evaluation across class-incremental, domain-incremental, and few-shot regimes demonstrates consistent improvements over prior methods in heterogeneous structured prediction settings. Our results expose fundamental failure modes of existing continual segmentation approaches and provide insight into learning robust dense predictors in dynamically evolving environments. Our code is available at https://github.com/prinshul/JASCL.git.

Real-world AI systems must continuously learn new things while facing changing data sources and limited labelled examples simultaneously. Existing continual learning research typically studies these challenges in isolation, leaving a critical gap. This paper formalizes "joint nonstationarity", where semantic classes, visual domains, and label availability all shift at once, reflecting practical deployment conditions. The authors propose JASCL, combining two mechanisms. First, Gradient Adaptive Stabilization injects parameter-wise noise scaled inversely to gradient magnitude, protecting important parameters from disruption while allowing flexible adaptation in less critical directions. Second, Prototype Anchored Supervision validates pseudo labels from unlabeled data using both prediction confidence and similarity to class prototypes computed from labelled examples, preventing compounding errors in self-training. Experiments spanning 3D medical segmentation and autonomous driving across 37 baselines show that existing methods, including large foundation models like SAM, collapse quickly under these conditions. JASCL maintains strong performance across multiple sessions with negligible computational overhead.