Geometric Convergence of Gauss–Newton for Neural Networks: Riemannian Geometry and Adaptive Damping

Ill-conditioned kernel matrices can make first-order methods for training neural networks converge slowly. We establish non-asymptotic convergence bounds for the Gauss–Newton method in both under- and overparameterized regimes, showing it avoids these conditioning bottlenecks. In the underparameterized setting, Gauss–Newton gradient flow in parameter space induces a Riemannian gradient flow on a low-dimensional submanifold of function space. Using tools from Riemannian optimization, we show that, under an appropriate output scaling, the loss satisfies geodesic Polyak–Lojasiewicz and Lipschitz-smoothness conditions, implying geometric convergence to the optimal in-class predictor at an explicit rate independent of Gram-matrix conditioning. In the overparameterized setting, we identify adaptive, curvature-aware regularization schedules and prove fast geometric convergence to a global optimum for both Gauss–Newton gradient flow and discrete-time Gauss–Newton iterates, with rates independent of the minimum eigenvalue of the neural tangent kernel. Overall, Gauss–Newton can be provably faster in ill-conditioned regimes where first-order methods slow down.

Training neural networks often takes many steps. One reason is that standard training methods can move slowly: they know which direction improves the model, but the geometry of the problem forces them to make only small progress at each step. We study a different training approach called the Gauss-Newton method, which uses more information about the shape of the problem than standard methods. For large models that can fit the training data perfectly, we show that Gauss-Newton can reach a perfect fit quickly by adapting to the curvature of the problem as training progresses. For smaller models that cannot fit the data perfectly, we show that Gauss-Newton still makes fast progress toward the best prediction the model can achieve. These results help explain why Gauss-Newton methods can avoid some slowdowns that affect standard training methods. This may guide the design of faster and more reliable ways to train neural networks, especially when standard training is inefficient.