Preserving Plasticity in Continual Learning via Dynamical Isometry

Continual training of deep neural networks under non-stationarity often leads to a progressive loss of plasticity, eventually limiting further learning. We relate plasticity to the empirical Neural Tangent Kernel, and identify dynamical isometry (the condition that layer-wise Jacobian singular values remain close to one) as a key mechanism for preserving plasticity in continual learning. We revisit a class of networks that are almost-everywhere isometric while remaining universal Lipschitz function approximators, demonstrating that near-dynamical isometry is compatible with expressive nonlinear representations. For general architectures, we propose an efficient isometry-promoting regularization scheme and identify a novel mechanism by which it can reactivate dormant ReLU units. Building on this, we introduce AdamO, an Adam-style adaptive optimizer that decouples isometry regularization from gradient updates, analogous to AdamW. We further reinterpret prior plasticity-preserving approaches through the lens of dynamical isometry, showing that they target only a partial measure of isometry. Across supervised and reinforcement-learning continual-learning benchmarks designed to induce plasticity loss, our methods consistently match or outperform existing approaches.

Neural networks usually learn one fixed task, but when they have to keep learning new things one after another, they often grow "stiff" and slowly lose the ability to pick up anything new — a problem called 'plasticity loss'. This paper traces that stiffening to a single underlying mathematical cause: as training goes on, the internal math that lets the network adjust its internal state gets scrambled, and over time, more and more tasks become impossible to learn. We propose a fix to this by keeping the network in a balanced state, where update signals pass through the network without being stretched or squashed — a property called "dynamical isometry". This significantly improves the plasticity of a neural network, effectively enabling networks to keep adapting to new problems over time.