Continuous-Time Piecewise-Linear Recurrent Neural Networks

In dynamical systems reconstruction (DSR) we aim to recover the dynamical system (DS) underlying observed time series. Specifically, we aim to learn a generative surrogate model which approximates the underlying, data-generating DS, and recreates its long-term properties (`climate statistics'). In scientific and medical areas, in particular, these models need to be mechanistically tractable -- through their mathematical analysis we would like to obtain insight into the recovered system's workings. Piecewise-linear (PL), ReLU-based RNNs (PLRNNs) have a strong track-record in this regard, representing SOTA DSR models while allowing mathematical insight by virtue of their PL design. However, all current PLRNN variants are discrete-time maps. This is in disaccord with the assumed continuous-time nature of most physical and biological processes, and makes it hard to accommodate data arriving at irregular temporal intervals. Neural ODEs are one solution, but they do not reach the DSR performance of PLRNNs and often lack their tractability. Here we develop theory for continuous-time PLRNNs (cPLRNNs): We present a novel algorithm for training and simulating such models, bypassing numerical integration by efficiently exploiting their PL structure. We further demonstrate how important topological objects like equilibria or limit cycles can be determined semi-analytically in trained models. We compare cPLRNNs to both their discrete-time cousins as well as Neural ODEs on DSR benchmarks, including systems with discontinuities which come with hard thresholds.

This work focuses on understanding complex systems—such as those in biology or physics—by learning models from time-series data that can reproduce how these systems behave over time. A key goal is not just accurate prediction, but also interpretability: scientists want models they can analyze to gain insight into how the underlying system works. Existing models based on piecewise-linear recurrent neural networks perform well and are interpretable, but they describe systems in discrete time steps, which is in disaccord with the fact that most real-world processes evolve continuously. This mismatch also makes it difficult to handle data collected at irregular time intervals. While continuous-time approaches like Neural ODEs exist, they often fall short in performance or interpretability. To address this, we introduce a new class of models that combine the strengths of both approaches. These models operate in continuous time while retaining a structure that allows for mathematical analysis. We develop efficient methods to train and simulate them without relying on costly numerical integration, and we show how key system behaviors—such as stable states or repeating patterns—can be identified analytically. Experiments demonstrate that our approach performs competitively on benchmark problems.