Skip to yearly menu bar Skip to main content


Poster
in
Workshop: HiLD: High-dimensional Learning Dynamics Workshop

Effects of Overparameterization on Sharpness-Aware Minimization: A Preliminary Investigation

Sungbin Shin · Dongyeop Lee · Namhoon Lee


Abstract:

Sharpness-aware minimization (SAM) has exhibited its effectiveness in increasing generalization performance. Despite the concurrent empirical and theoretical evidence of its validity, however, there is little work that studies the effect of overparameterization on SAM. In this work, we experiment to observe how SAM changes its behavior with respect to different settings of overparameterization including sparsity. We find that SAM provides a larger benefit over stochastic gradient descent (SGD) in overparameterized regimes and low-sparsity settings. Furthermore, we discover that SAM shows different behaviors on large sparse models compared to their small dense counterparts despite having a similar number of parameters.

Chat is not available.