Understanding SAM through Minimax Perspective

Sharpness-Aware Minimization (SAM) empirically boosts generalization by seeking parameters that minimize the worst-case loss in a small neighborhood, yet existing theory explains its behavior under either Polyak-Lojasiewicz (PL) condition or upper bounded perturbation radius. We revisit SAM through the bilevel minimax problem $\min_{\theta}\max_{\|\Delta\|\le\rho}l(\theta+\Delta)$ and derive a $(\theta,\Delta)$ gradient flow ODE whose equilibria coincide with the problem’s optimality conditions. A Lyapunov argument-free of convexity assumptions, quantifies how the optimality gap depends on the radius~$\rho$ and local curvature. Discretizing the flow yields a \emph{Multi-step SAM} algorithm that recovers classical SAM as $\rho\to 0$. Moreover, our analysis and the resulting algorithm remain valid even for large $\rho$, providing guidance for aggressive neighborhood exploration. Experiments on synthetic objectives and CIFAR-10 validate the predicted gains from multiple inner updates, bridging the gap between SAM’s minimax intuition and its practical implementation.

Machine learning models often memorize data instead of learning true patterns, getting trapped in fragile "sharp valleys" in the optimization landscape. To fix this, algorithms play a "min-max" game: they try to minimize error while a hypothetical adversary tries to maximize it by pushing the model to the worst nearby position. However, standard methods rush this game by guessing the adversary's move in just one step, which frequently causes the algorithm to miscalculate the terrain and stall. To solve this, we mathematically modeled this min-max game using a two-timescale continuous framework: imagining two clocks running at different speeds. The "fast clock" allows the adversary to take multiple quick steps to accurately find the true worst-case bump. Meanwhile, the "slow clock" uses this accurate map to safely guide the main model down into a wide, stable valley. Based on this theory, we designed a practical method that periodically uses these fast, multi-step explorations to navigate difficult terrain. By properly separating the min-max dynamics, our approach successfully bypasses deceptive traps. This ensures AI systems reliably find robust solutions and generalize better to new, unseen data.