Balancing Plasticity and Stability with Fast and Slow Successor Features

A hallmark of intelligence is the ability to adapt in non-stationary environments, yet deep Reinforcement Learning (RL) agents often struggle in such settings. Prior studies introduce non-stationarity through abrupt shifts in features or dynamics, whereas real-world environments often evolve gradually through continual drift. This distinction has important implications for the ``stability-plasticity dilemma'' in RL, as abrupt task changes may demand more plasticity than naturalistic settings. To address this, we modify existing 3D Miniworld and MuJoCo environments to incorporate naturalistic, continual non-stationarity, and use them to examine how stability and adaptation affect performance under continuous environmental change. We find that methods favoring stability, such as synaptic consolidation, outperform approaches focused on plasticity, such as parameters resetting. Motivated by this result, and prior evidence that Successor Features (SFs) reduce interference, we investigate whether SFs are better consolidation targets than Q-values. Across both environments, applying neuro-inspired synaptic consolidation to SFs yields superior performance on continually changing settings. Moreover, consolidation is most effective when SFs are stabilized across multiple timescales, which capture complementary aspects of gradual environmental change. Together, these results suggest that stability is more critical in continual learning when changes are gradual, and that multi-timescale consolidation of predictive representations is an effective approach.

Many real-world environments evolve gradually and continuously over time, such as changing weather conditions or shifts in biological systems. When environments change continuously, do AI agents primarily struggle because they cannot learn new information, or because they cannot retain previously acquired knowledge? Surprisingly, we found that under gradual and continuous change, the primary challenge is not a loss of plasticity—the ability to learn new information—but a loss of stability: the ability to preserve useful knowledge while continuing to adapt. Inspired by biological memory systems, we developed a multi-timescale learning approach that combines fast adaptation with slow, long-term memory. We found that stabilizing predictive representations of the environment, rather than task-specific value estimates, leads to more robust learning. Our findings suggest that building AI systems capable of lifelong adaptation may require preserving predictive knowledge across multiple timescales, providing a path toward more reliable learning in continuously changing real-world environments.