Quantization and the Bottom of the Loss Landscape

We introduce two physics-inspired methods for the compression of neural networks that encourageweight clustering, in anticipation of model quantization, by adding attractive interactions betweenparameters to the loss. Our two methods implement interactions either directly or via an intermediaryset of centroids. By applying these methods to pre-trained neural networks, we investigate theexistence of compressible configurations near the bottom of the loss landscape. The direct interactionapproach suggests the existence of multiple, qualitatively distinct compressed configurations closeto pre-trained models, and the centroid-mediated approach provides a pipeline for quantization thatis competitive with extant quantization methods.