Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization

The remarkable generalization properties of overparameterized networks are often attributed to implicit biases, such as norm minimization at small learning rates and low sharpness in the Edge-of-Stability regime. In this work, we argue that a comprehensive understanding of the generalization performance of gradient descent requires analyzing the interaction between these various forms of implicit regularization. We empirically demonstrate that the learning rate interpolates between low parameter norm and low sharpness of the trained model. We furthermore prove that neither implicit bias alone minimizes the generalization error for diagonal linear networks trained on a simple regression task. These findings demonstrate that focusing on a single implicit bias is insufficient to explain good generalization, and they motivate a broader view of implicit regularization that captures the dynamic trade-off between norm and sharpness induced by non-negligible learning rates.

Neural networks often have many ways to fit the training data, but some of these solutions work much better on new, unseen data than others. A central question is why the training process tends to find such useful solutions. Existing explanations point to different hidden preferences of the training algorithm: it may favor models with smaller parameters, or it may favor flatter solutions whose performance is less sensitive to small changes in the model. These preferences do not always agree, and in some cases they point to very different solutions. We study this tension through experiments and through simplified mathematical models. Our results suggest that, in several settings, neither preference alone identifies the best solution; instead, good performance can depend on a trade-off between them. This gives a broader view of why neural networks generalize: the training process may combine several competing preferences rather than follow a single simple principle.