Effects of Overparameterization on Sharpness-Aware Minimization: A Preliminary Investigation

Sharpness-aware minimization (SAM) has exhibited its effectiveness in increasing generalization performance. Despite the concurrent empirical and theoretical evidence of its validity, however, there is little work that studies the effect of overparameterization on SAM. In this work, we experiment to observe how SAM changes its behavior with respect to different settings of overparameterization including sparsity. We find that SAM provides a larger benefit over stochastic gradient descent (SGD) in overparameterized regimes and low-sparsity settings. Furthermore, we discover that SAM shows different behaviors on large sparse models compared to their small dense counterparts despite having a similar number of parameters.