Sharpness-Aware Minimization Can Hallucinate Minimizers

Sharpness-Aware Minimization (SAM) is widely used to seek flatter minima—often linked to better generalization. In its standard implementation, SAM updates the current iterate using the loss gradient evaluated at a point perturbed by distance $\rho$ along the normalized gradient direction. We show that, for some choices of $\rho$, SAM can stall at points where this shifted (perturbed-point) gradient vanishes despite a nonzero original gradient, and therefore, they are not stationary points of the original loss. We call these points hallucinated minimizers, prove their existence under simple nonconvex landscape conditions (e.g., the presence of a local minimizer and a local maximizer), and establish sufficient conditions for local convergence of the SAM iterates to them. We corroborate this failure mode in neural network training and observe that it aligns with SAM's performance degradation often seen at large $\rho$. Finally, as a practical safeguard, we find that a short initial SGD warm-start before enabling SAM mitigates this failure mode and reduces sensitivity to the choice of $\rho$.

When researchers train AI models, they want them to work not only on examples they practiced on but also on new ones. A popular training method, Sharpness-Aware Minimization (SAM), helps by checking whether a model still performs well after its internal settings are nudged slightly. This usually encourages more reliable solutions. Our paper shows that the standard way SAM is implemented can backfire when the nudge is too large. In that case, SAM can be fooled into stopping at a false destination. We call this a "hallucinated minimizer": SAM sees no reason to keep learning, even though the model has not actually reached a good solution for the original training task. We give a mathematical explanation for when such false destinations can appear, and we show matching warning signs in neural-network experiments. We also provide a simple safeguard: begin with a short period of standard training, and turn on SAM only afterward. These results help researchers and practitioners use SAM more reliably, with less risk that an overly large perturbation derails training.