On the Power of Statistics in Class-Incremental Learning with Pretrained Models

Recent class-incremental learning (CIL) methods built on large pre-trained vision models have shown that strong performance can be retained even under strict data access constraints. This raises a fundamental question: which properties of pre-trained representations make such recovery possible in the class-incremental setting? In this work, we show that class-level feature statistics play a central role in enabling effective CIL under strong pre-training. When the visual backbone is frozen, maintaining simple class-wise statistical estimators of features can recover a substantial fraction of the performance achieved by static joint training across diverse benchmarks. We make this observation explicit through deliberately minimal reference points built on frozen CLIP representations. In particular, we demonstrate that competitive performance can be achieved without continual parameter updates, by performing class-incremental inference based solely on class-level statistical estimators instantiated from frozen features. Our findings suggest that class-level statistics constitute an important and previously underemphasized component of recent PTM-based CIL approaches, offering a complementary perspective for understanding their strong empirical performance. Our code is available at https://github.com/HdTgon/baseCIL.

Modern AI systems can learn to recognize new categories over time, even when they no longer have access to previously seen training data. However, it is still unclear why today’s large pre-trained vision models are so effective at this type of continual learning. In this work, we show that much of this ability comes from simple statistical patterns in the image features produced by the pre-trained model. By keeping the original model fixed and only storing lightweight summaries for each category, our method can still recognize new classes surprisingly well without repeatedly retraining the model. These findings provide a simpler explanation for why large pre-trained models perform so strongly in continual learning settings, and may help guide the design of more efficient learning systems in the future.