Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning

We investigate why deep neural networks suffer from loss of plasticity in continual learning, and thus fail to learn new tasks without reinitializing parameters. We show that this failure is preceded by Hessian spectral collapse at new-task initialization, where meaningful curvature directions vanish and gradient descent becomes ineffective. Analyzing a linearized ReLU network, we derive explicit $\epsilon$-rank conditions for successful training and prove that the loss-weighted Gram matrix is spectrally equivalent to the Generalized Gauss-Newton approximation, thereby relating NTK dynamics to Hessian curvature. Targeting spectral collapse directly, we then discuss the Kronecker factored approximation of the Hessian, which motivates two regularization enhancements: maintaining high effective feature rank and applying L2 penalties. Experiments on continual supervised and reinforcement learning tasks confirm that combining these two regularizers effectively preserves plasticity.

We investigate why neural networks fail to adapt to new tasks continually without the ability to reset. We find that this pathology, called loss of plasticity, is preceded by Hessian spectral collapse. This is where a matrix, called the Hessian, which contains key information about a neural network, becomes overly simplified. We analyse a simple model of a neural network and propose a fix that keeps this Hessian matrix healthy. We show that our method is effective on many different continual problems with many different kinds of neural networks.