Oral
Why do Larger Models Generalize Better? A Theoretical Perspective via the XOR Problem
Alon Brutzkus · Amir Globerson

Thu Jun 13th 11:00 -- 11:20 AM @ Grand Ballroom
in Deep Learning Theory »

Empirical evidence suggests that neural networks with ReLU activations generalize better with over-parameterization. However, there is currently no theoretical analysis that explains this observation. In this work, we provide theoretical and empirical evidence that, in certain cases, overparameterized convolutional networks generalize better than small networks because of an interplay between weight clustering and feature exploration at initialization. We demonstrate this theoretically for a 3-layer convolutional neural network with max-pooling, in a novel setting which extends the XOR problem. We show that this interplay implies that with overparamterization, gradient descent converges to global minima with better generalization performance compared to global minima of small networks. Empirically, we demonstrate these phenomena for a 3-layer convolutional neural network in the MNIST task.

Author Information

Alon Brutzkus (Tel Aviv University)
Amir Globerson (Tel Aviv University, Google)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors