Efficient Continuous-Depth Modeling with GRU Equivalents

Continuous-Depth Neural Networks (CDNNs), including Neural Ordinary Differential Equations (ODEs) and Liquid-Time-Constant Neural Networks (LTC-NN), suffer from high computational costs due to solving numerous nonlinear ODEs during training and inference. We introduce Continuous Depth Acceleration (CoDA), a framework that leverages Mori–Zwanzig/Koopman operator theory to replace continuous-depth layers requiring multiple nonlinear ODEs with a compact GRU module, a single low-dimensional linear ODE, and a dense layer. We prove PAC learnability of CoDA, establishing that this transformation preserves accuracy and can be applied repeatedly across multiple layers with unified backpropagation. Experiments on the Liquid Foundation Model (LFM-1.2B) demonstrate $6.7\times$ training speedup and $1.8\times$ inference speedup without loss of accuracy. Across six real-world LTC-NN applications, CoDA consistently outperforms state-of-the-art acceleration techniques—including neural flows, model order reduction, and variational formulations—in both training and inference time while maintaining competitive or superior accuracy. The implementation and datasets are publicly available at \url{https://github.com/ImpactLabASU/CoDA-ICML2026}.

Modern AI systems are often built from neural networks—flexible pattern-recognition models that learn from data. A particularly versatile family of these models, called continuous-depth networks, treats information as flowing smoothly through time rather than jumping through fixed steps. This flexibility lets them adapt naturally to streaming data like sensor readings, voice, or video, and they form the basis of recent "liquid" foundation models. The cost of this flexibility is speed: running or training these models requires solving a long sequence of small mathematical problems one after another, which is slow and energy-hungry, especially on phones, watches, and other small devices. We developed a method called CoDA that makes these models dramatically faster without changing what they can do. The key idea, borrowed from physics, is that the complicated step-by-step behavior of a continuous-depth network can be replaced by a much smaller, simpler calculation that produces the same answer in a single shot. We prove mathematically that this replacement is reliable—the accelerated model reaches the same accuracy as the original—and we show how to plug it into existing systems without retraining from scratch.