Continual Learning of Physical Systems via Derivative Distillation

Continual learning (CL) designs models that adapt to non-stationary data streams while preserving previously acquired knowledge. Moving away from current CL benchmarks and datasets, we turn our attention to scientific machine learning, and we study how CL can approach physical systems, which are inherently continuous in time and space and are naturally described by differential equations over spatio-temporal domains. We adopt Derivative Distillation, a distillation-based approach that leverages model derivatives as a compact representation of knowledge and integrates it within a physics-informed learning framework. Our results show that the Derivative Distillation enables stable adaptation with minimal forgetting. In some cases, the performance surpasses that of a model jointly trained on all data at once. These findings highlight physical systems as a promising and realistic benchmark for CL.