Self-Monitoring Warmup: Data-Driven Confidence Rectification for PINN Optimizers

Adaptive optimizers such as Adam, RAdam, and SOAP decide how aggressively to trust their adaptive learning rate from a fixed rule: a closed-form factor, an iteration-count schedule, or a distributional assumption on gradients. In physics-informed neural networks (PINNs), where gradients couple multi-scale IC, BC, and PDE-residual terms through differential operators, these assumptions fit poorly with the observed training dynamics: gradients are heterogeneous across parameters, and Adam- and SOAP-style updates exhibit early-training gradient distortion. We argue that the optimizer should set its rectification from the empirical statistics of its own second-moment estimates, rather than from the iteration count, and propose \emph{Adaptive Confidence Rectification} (\ourmethod): a data-driven warmup in which each parameter's rectification is set by the empirical volatility of its second-moment estimate, turned into a bounded per-parameter confidence. \ourmethod makes no assumption on the gradient distribution and acts independently on each parameter. We prove \ourmethod strictly reduces the variance of the effective learning rate over RAdam and contracts the step under low confidence. The method is general and can be combined with any adaptive optimizer; we demonstrate it on RAdam, SOAP, Sophia, and AdaHessian, where it consistently improves performance across four PDEs (Allen--Cahn, Wave, KdV, Burgers) and three model scales.