Quantifying Symmetries: How Optimisers Impact the Functional Dimension

Overparameterised neural networks exhibit extensive symmetries in the parameter space. We investigate which optimisers are implicitly biased towards parameter solutions with a larger number of hidden symmetries. We provide both theoretical results, and study this in regression experiments by evaluating the functional dimension (the number of independent directions in the pa- rameter space along which the network changes), that serves as an empirical tool to evaluate the prevalence of hidden symmetries. Moreover, we relate our results on functional dimension to flatness metrics involving the Hessian of the loss with respect to the parameters.