On the Theory of Continual Learning with Gradient Descent for Neural Networks

Continual learning, the ability of a model to adapt to an ongoing sequence of tasks without forgetting earlier ones, is a central goal of artificial intelligence. To better understand its underlying mechanisms, we study the limitations of continual learning in a tractable yet representative setting. Specifically, we analyze one-hidden-layer quadratic neural networks trained by gradient descent on a sequence of XOR-cluster datasets with Gaussian noise, where different tasks correspond to clusters with orthogonal means. Our analysis is based on a tight characterization of gradient descent dynamics for the training loss, which yields explicit bounds on the rate of train-time forgetting as functions of the number of iterations, sample size, number of tasks, and hidden-layer width. We then leverage an algorithmic stability framework to bound the generalization gap, leading to corresponding guarantees on test-time forgetting. Together, our results provide the first closed-form guarantees for forgetting in continual learning with neural networks and show how key problem parameters jointly govern forgetting dynamics. Numerical experiments corroborate our theoretical results.

Artificial intelligence systems are often trained in stages: they learn one task first, and later they are updated using new data. A major problem is that learning something new can often cause the system to forget what it learned before. This is known as catastrophic forgetting, and it is one of the main challenges in building AI systems that can keep learning over time. In this work, we study forgetting in a simplified and tractable setting: a neural network trained one task at a time using gradient descent. We analyze how the network’s parameters change as new tasks arrive, and we derive explicit formulas showing how forgetting depends on the number of training examples, the network size, the number of tasks, and the training time. Our results show that, in this setting, forgetting can become small when the network is wide enough, later tasks have enough data, and training is stopped at the right time. We also show that a common form of regularization behaves mainly like using a smaller learning rate, rather than fundamentally changing the forgetting behavior. These findings help explain when simple sequential training can work, and they provide theoretical guidance for designing AI systems that learn continually without losing earlier knowledge.