Revisiting the Volume Hypothesis

Modern deep neural networks often contain far more parameters than needed to fit their training data, yet they achieve impressive generalization. A common explanation for this success is the implicit bias of stochastic gradient descent (SGD). An alternative volume hypothesis posits that, within low training-loss regions, loss-landscape basins leading to strong generalization occupy much larger regions of weight space than basins that generalize poorly, and therefore SGD is simply more likely to land in the former. Recent experimental explorations of this idea present seemingly contradictory results. While in one set of experiments randomly sampling the network weights until achieving zero training error yielded poor generalization, molecular dynamics density estimates supported the volume hypothesis. We observe that these experiments were performed at different dataset size regimes, and explore an intermediate regime using the Replica Exchange Wang–Landau algorithm to estimate the joint density of states over training and test accuracies in binary networks. Across several architectures and datasets, we show that the generalization advantage of gradient learning over random sampling training generally diminishes as the training data size grows, suggesting a resolution of the paradox.

Deep neural networks often have many more parameters than they need to memorize their training data, yet they can still perform well on new, unseen examples. One popular explanation is that the training algorithm, stochastic gradient descent, has a built-in tendency to find solutions that generalize well. Another possibility is the “volume hypothesis”: among all the ways a network can fit the training data, the good solutions simply occupy much larger regions of parameter space than the bad ones, making them easier to find. Previous experiments seemed to give conflicting answers. Some showed that randomly searching for networks that perfectly fit the training data led to poor generalization, while others suggested that good-generalizing solutions do take up more volume. We argue that these experiments looked at different amounts of training data. To bridge this gap, we study an intermediate regime using a specialized sampling method in simplified binary neural networks. Our results show that as the amount of training data increases, the difference between gradient-based training and random sampling becomes smaller. This suggests that the apparent contradiction can be explained by dataset size: when enough training data is used, good-generalizing solutions may indeed occupy a larger portion of the space of low-training-error networks.