Unveiling Multi-regime Patterns in SciML: Distinct Failure Modes and Regime-specific Optimization

Neural networks trained under different hyperparameter settings can fall into distinct training "regimes," with consistent behavior within regimes and qualitative differences across regimes. In this paper, we study such multi-regime behavior in scientific machine learning (SciML) models through a regime-aware diagnostic framework that jointly analyzes performance, training dynamics, and loss-landscape geometry. We identify three key findings: (i) a consistent three-regime structure emerges across many standard SciML models, different constraint enforcements, and various optimizer designs; (ii) optimization effectiveness is regime-specific, with no single method performing well across all regimes; and (iii) SciML models can exhibit fine-grained failure modes that can challenge conventional interpretations of standard loss-landscape metrics. Our results provide an approach to establish a unified, task-oblivious perspective on failure modes in SciML and to inform regime-aware guidance for improving robustness. We validate these findings across widely-used SciML models, including physics-informed neural networks, neural operators, and neural ordinary differential equations, on benchmarks spanning representative ordinary and partial differential equations.

Scientific machine learning (SciML) combines neural networks with physical laws to model complex systems. Although these methods are promising, they are often difficult to train reliably: small changes in settings, such as data size/optimization/physics, can suddenly cause performance to improve, stagnate, or fail. Understanding why this happens remains a major challenge. In this work, we show that many popular SciML models consistently fall into three distinct "training regimes," each with its own optimization behavior and failure patterns. We studied a wide range of models, including physics-informed neural networks, neural operators, and neural differential equation models, and found that no single optimization strategy works well across all regimes. We also discovered that common tools used to diagnose neural network training can give misleading signals in SciML. For example, models can appear to fit the data well while still failing to learn the correct physical behavior. By analyzing training dynamics and loss landscapes together, our work provides a more unified way to understand why SciML models succeed or fail. These findings can help researchers design more reliable and robust scientific AI systems for real-world scientific and engineering applications.