Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise

Approximate second-order optimizers are increasingly showing promise in accelerating training of deep learning models, yet their practical performance depends critically on how preconditioning is applied. Two predominant approaches to preconditioning are based on (1) Adam, which leverages statistics of the current gradient, and (2) Gauss-Newton (GN) methods, which use approximations to the Fisher information matrix (often raised to a power). This work compares these approaches through the lens of two key factors: the choice of basis in the preconditioner and the impact of gradient noise from mini-batching. To gain insights, we analyze these optimizers on quadratic objectives and logistic regression under all four quadrants. We show that regardless of the basis, there exist instances where Adam outperforms both $\text{GN}^{-1}$ and $\text{GN}^{-1/2}$ in full-batch settings. Conversely, in the stochastic regime, Adam behaves similarly to $\text{GN}^{-1/2}$ under a Gaussian data assumption. These theoretical results are supported by empirical studies on both convex and non-convex objectives.