The Importance of Being Lazy: Scaling Limits of Continual Learning

Despite recent efforts, neural networks still struggle to learn in non-stationary environments, and our understanding of catastrophic forgetting (CF) is far from complete.In this work, we perform a systematic study on the impact of model scale and the degree of feature learning in continual learning. We reconcile existing contradictory observations on scale in the literature, by differentiating between lazy and rich training regimes through a variable parameterization of the architecture. We show that increasing model width is only beneficial when it reduces the amount of feature learning, yielding more laziness. Using the framework of dynamical mean field theory, we then study the infinite width dynamics of the model in the feature learning regime and characterize CF, extending prior theoretical results limited to the lazy regime. We study the intricate relationship between feature learning, task non-stationarity, and forgetting, finding that high feature learning is only beneficial with highly similar tasks. We identify a transition modulated by task similarity where the model exits an effectively lazy regime with low forgetting to enter a rich regime with significant forgetting. Finally, our findings reveal that neural networks achieve optimal performance at a critical level of feature learning, which depends on task non-stationarity and transfers across model scales. This work provides a unified perspective on the role of scale and feature learning in continual learning.

Neural networks are powerful learners when their tasks stay consistent, but they struggle when tasks change, often losing previously learned information—a problem known as catastrophic forgetting. Although catastrophic forgetting has been studied for a long time, it’s still unclear how best to prevent it. For instance, we don’t know if simply making networks larger helps them remember better, as research findings have been mixed.We systematically studied how network size and their level of adaptability—called “feature learning”—affect their ability to continuously remember changing tasks. Networks can be “lazy,” meaning they minimally adjust their internal settings, or “rich,” meaning they extensively modify themselves. Our experiments show that making networks larger only helps if they remain relatively “lazy.” Networks that extensively adapt their internal settings rapidly forget previously learned tasks, especially if the tasks are very different, and making these networks larger doesn’t prevent this forgetting.Our results demonstrate that maintaining some degree of laziness is essential for neural networks to retain knowledge over time. Practically, this insight helps researchers and engineers design better neural networks for real-world applications where tasks frequently change, such as adaptive robotics or lifelong learning systems.