Towards Understanding Adam Convergence on Highly Degenerate Polynomials

Adam is a widely used optimization algorithm in deep learning, yet the specific class of objective functions where it exhibits inherent advantages remains underexplored. Unlike prior studies requiring external schedulers and $\beta_2$ near 1 for convergence, this work investigates the ``natural'' auto-convergence properties of Adam. We identify a class of highly degenerate polynomials where Adam converges automatically without additional schedulers. Specifically, we derive theoretical conditions for local asymptotic stability on degenerate polynomials and demonstrate strong alignment between theoretical bounds and experimental results. We prove that Adam achieves local linear convergence on these degenerate functions, significantly outperforming the sub-linear convergence of Gradient Descent and Momentum. This acceleration stems from a decoupling mechanism between the second moment $v_t$ and squared gradient $g_t^2$, which exponentially amplifies the effective learning rate. Finally, we characterize Adam's hyperparameter phase diagram, identifying three distinct behavioral regimes: stable convergence, spikes, and SignGD-like oscillation.

Training deep learning models is essentially like navigating a complex, hilly landscape to find the lowest valley. While the Adam optimizer is the most popular "vehicle" for this journey, researchers still don't fully understand exactly where and why it naturally outperforms older methods. We answer this by studying Adam's behavior when it encounters highly "degenerated" valleys—areas of the landscape so pathologically flat that traditional algorithms lose all sense of direction and stall. Our paper presents the exciting result that Adam possesses a built-in "auto-acceleration" mechanism when navigating these extremely flat, degenerate regions. Traditionally, engineers thought algorithms needed manual speed controllers to escape such areas—akin to constantly shifting gears to get out of a flat, muddy plain. Instead, we show Adam naturally separates its memory of past progress from immediate obstacles, speeding up exactly when older methods crawl. We also map out how Adam's starting settings dictate its behavior to avoid erratic bouncing. Our findings reveal that these degenerate, flat directions are actually a crucial key to understanding Adam's acceleration over traditional methods like Stochastic Gradient Descent (SGD). Ultimately, this provides a strong foundation for designing even better optimizers in the future and deepens our understanding of how network architectures interact with their complex training landscapes.