Timezone: »
The enormous size of modern deep neural net-works makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an increasingly important task. Tremendous advances has been made recently, where the main technical building blocks are pruning, quantization, and low-rank factorization. In this paper, we propose principled approaches to improve upon the common heuristics used in those building blocks, by studying the fundamental limit for model compression via the rate distortion theory. We prove a lower bound for the rate distortion function for model compression and prove its achievability for linear models. Although this achievable compression scheme is intractable in practice, this analysis motivates a novel objective function for model compression, which can be used to improve classes of model compressor such as pruning or quantization. Theoretically, we prove that the proposed scheme is optimal for compressing one-hidden-layer ReLU neural networks. Empirically,we show that the proposed scheme improves upon the baseline in the compression-accuracy tradeoff.
Author Information
Weihao Gao (University of Illinois at Urbana-Champaign)
Yu-Han Liu (Google)
Chong Wang (ByteDance Inc.)
Sewoong Oh (University of Washington)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Oral: Rate Distortion For Model Compression:From Theory To Practice »
Tue. Jun 11th 10:15 -- 10:20 PM Room Room 102
More from the Same Authors
-
2021 : Robust and Differentially Private Covariance Estimation »
Logan Gnanapragasam · Jonathan Hayase · Sewoong Oh -
2019 Poster: Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms »
Ashok Vardhan Makkuva · Pramod Viswanath · Sreeram Kannan · Sewoong Oh -
2019 Oral: Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms »
Ashok Vardhan Makkuva · Pramod Viswanath · Sreeram Kannan · Sewoong Oh -
2017 Poster: Sequence Modeling via Segmentations »
Chong Wang · Yining Wang · Po-Sen Huang · Abdelrahman Mohammad · Dengyong Zhou · Li Deng -
2017 Talk: Sequence Modeling via Segmentations »
Chong Wang · Yining Wang · Po-Sen Huang · Abdelrahman Mohammad · Dengyong Zhou · Li Deng