Heavy-tailed Physics-Informed Neural Networks

Physics-informed neural networks (PINNs) enforce physical laws by minimizing partial differential equation (PDE) residuals and auxiliary constraints. Standard training relies on a mean-squared error (MSE) objective, which implicitly assumes independent Gaussian residuals with a fixed global variance. We show theoretically and empirically that residuals encountered during PINN training are heterogeneous and heavy-tailed, revealing a systematic mismatch with this assumption. As a consequence, a small number of large residuals can disproportionately dominate both the loss and gradient, leading to poorly balanced optimization dynamics. Motivated by this mismatch, we adopt a Student-$t$ residual model to explicitly capture heavy-tailed behavior. An equivalent hierarchical representation yields an expectation–maximization (EM) algorithm that alternates between estimating residual-dependent weights and optimizing network parameters via a weighted MSE objective, allowing existing PINN solvers to be reused in the M-step. The resulting training dynamics bound the influence of extreme residuals and admit almost sure convergence guarantees under standard stochastic optimization assumptions. Experiments across a diverse suite of challenging PDE benchmarks demonstrate consistently improved solution accuracy and robustness compared to standard PINN training.

Physics-informed neural networks, or PINNs, are designed to solve physics problems by learning from both data and the equations that describe the underlying system. In standard PINN training, all errors are treated using a squared-error loss. This means that very large errors receive much more attention than smaller ones. That can be problematic because, during training, PINNs often produce errors that are uneven and sometimes extreme. A few difficult points can therefore dominate the learning process and make the model train poorly. This work proposes a more robust way to train PINNs. Instead of assuming that the training errors behave nicely, it models them with a Student-t distribution, which is better at handling occasional large errors. The resulting method automatically adjusts the importance of each error: ordinary errors are still used, but unusually large ones are prevented from overwhelming the training. The approach can be implemented as a weighted version of the usual PINN loss, so existing PINN training methods can still be used. The paper also provides theoretical support for the method and shows, across several difficult physics benchmarks, that it improves both accuracy and reliability compared with standard PINN training.