Non-Euclidean Gradient Descent Operates at the Edge of Stability

The Edge of Stability (EoS) is a phenomenon where the sharpness (largest eigenvalue) of the Hessian approaches and then hovers near the stability threshold $2/\eta$ during gradient descent (GD) with step size $\eta$. Despite (apparently) violating classical smoothness assumptions, EoS has been widely observed in deep learning, but its theoretical foundations remain incomplete. We provide an interpretation of EoS through the lens of Directional Smoothness [Mishkin et al., 2024]. This interpretation naturally extends to non-Euclidean norms, which we use to define generalized sharpness under an arbitrary norm. Our generalized sharpness measure includes previously studied vanilla GD and preconditioned GD as special cases, as well as methods for which EoS has not been studied, such as $\ell_{\infty}$-descent, Block CD, Spectral GD, and their normalized versions. Through experiments on neural networks, we show that non-Euclidean GD with our generalized sharpness also exhibits progressive sharpening followed by oscillations around or above the threshold $2/\eta$. Practically, our framework provides a geometry-aware spectral diagnostic that can be applied across a broad class of non-Euclidean gradient methods.

Classical optimization literature does not fully capture the dynamics of gradient descent when training neural networks. In particular, theory often predicts that training should become unstable when the loss landscape becomes too “sharp,” but neural networks frequently operate near this predicted instability boundary without diverging. This behavior is known as the Edge of Stability. This work extends the study of Edge of Stability beyond vanilla gradient descent. It introduces a generalized notion of sharpness that depends on the geometry used by the optimizer, covering methods such as non-Euclidean descent, block coordinate descent, and spectral gradient descent. The paper shows, both theoretically and experimentally, that these methods also exhibit progressive sharpening followed by oscillations near the stability threshold. This provides a unified way to understand stability across different optimizers and helps explain why neural network training can remain effective in regimes where classical theory falls short.