Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization

Sharpness-Aware Minimization (SAM) improves generalization by minimizing the worst-case loss within a fixed parameter-space radius neighborhood. SAM and its variants mainly rely on a first-order linearized surrogate, while flat minima are inherently a second-order (curvature) notion. We revisit this mismatch and propose Loss-Equated SAM (LE-SAM), which inverts the traditional SAM mechanism that replaces the fixed perturbation radius with a fixed loss-space budget, effectively removing gradient-norm–dominated learning signals and shifting optimization toward curvature-dominated terms. Extensive experiments across diverse benchmarks and tasks demonstrate the strong generalization ability of LE-SAM that consistently outperforms SAM and even its variants, achieving state-of-the-art performance.

Modern deep neural networks often perform very well on training dataset but may fail to work on unseen data. Existing methods usually disturb the model by a fixed amount, but this can make training focus too much on the size of the current training signal rather than on whether the solution is truly stable. This paper proposes a simple change: instead of fixing how much we disturb the model, we fix how much the model’s error is allowed to increase, and then choose the disturbance accordingly. This helps the model learn more stable solutions.