Learn Singularly Perturbed Solutions via Homotopy Dynamics

Solving partial differential equations (PDEs) using neural networks has become a central focus in scientific machine learning. Training neural networks for singularly perturbed problems is particularly challenging due to certain parameters in the PDEs that introduce near-singularities in the loss function. In this study, we overcome this challenge by introducing a novel method based on homotopy dynamics to effectively manipulate these parameters. From a theoretical perspective, we analyze the effects of these parameters on training difficulty in these singularly perturbed problems and establish the convergence of the proposed homotopy dynamics method. Experimentally, we demonstrate that our approach significantly accelerates convergence and improves the accuracy of these singularly perturbed problems. These findings present an efficient optimization strategy leveraging homotopy dynamics, offering a robust framework to extend the applicability of neural networks for solving singularly perturbed differential equations.

Solving partial differential equations (PDEs) is fundamental to modeling physical systems, yet traditional numerical methods can be computationally expensive and often break down on complex tasks—especially high-dimensional PDEs or large families of similar problems. Neural networks offer an attractive alternative; however, they frequently fail to converge on singularly perturbed PDEs, where small parameters induce sharp layers or rapid oscillations in the solution. Such behavior produces highly nonconvex loss landscapes, making it difficult for gradient-based optimizers to drive the loss down and easy to become trapped in poor local minima.To address this, we introduce a novel training strategy called Homotopy Dynamics. Our method gradually transforms an easier problem into the target problem by smoothly changing the PDE parameter during training. This helps the neural network stay close to the correct solution path and avoid poor local minima.We analyze this approach theoretically and show that it improves convergence. Empirically, we test it on several difficult problems, including the Allen–Cahn and Helmholtz equations, as well as in an operator learning setting with DeepONet. In all cases, our method significantly improves training stability and solution quality.This work not only deepens our understanding of training dynamics for neural PDE solvers, but also provides a practical tool for accelerating scientific computing with machine learning.