Timezone: »

 
Poster
Rate Distortion For Model Compression:From Theory To Practice
Weihao Gao · Yu-Han Liu · Chong Wang · Sewoong Oh

Tue Jun 11 06:30 PM -- 09:00 PM (PDT) @ Pacific Ballroom #169
in Posters Tue »

The enormous size of modern deep neural net-works makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an increasingly important task. Tremendous advances has been made recently, where the main technical building blocks are pruning, quantization, and low-rank factorization. In this paper, we propose principled approaches to improve upon the common heuristics used in those building blocks, by studying the fundamental limit for model compression via the rate distortion theory. We prove a lower bound for the rate distortion function for model compression and prove its achievability for linear models. Although this achievable compression scheme is intractable in practice, this analysis motivates a novel objective function for model compression, which can be used to improve classes of model compressor such as pruning or quantization. Theoretically, we prove that the proposed scheme is optimal for compressing one-hidden-layer ReLU neural networks. Empirically,we show that the proposed scheme improves upon the baseline in the compression-accuracy tradeoff.

Keywords: Algorithms · Information Theory and Estimation

Author Information

Weihao Gao (University of Illinois at Urbana-Champaign)
Yu-Han Liu (Google)
Chong Wang (ByteDance Inc.)
Sewoong Oh (University of Washington)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors