Timezone: »
The enormous size of modern deep neural net-works makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an increasingly important task. Tremendous advances has been made recently, where the main technical building blocks are pruning, quantization, and low-rank factorization. In this paper, we propose principled approaches to improve upon the common heuristics used in those building blocks, by studying the fundamental limit for model compression via the rate distortion theory. We prove a lower bound for the rate distortion function for model compression and prove its achievability for linear models. Although this achievable compression scheme is intractable in practice, this analysis motivates a novel objective function for model compression, which can be used to improve classes of model compressor such as pruning or quantization. Theoretically, we prove that the proposed scheme is optimal for compressing one-hidden-layer ReLU neural networks. Empirically,we show that the proposed scheme improves upon the baseline in the compression-accuracy tradeoff.
Author Information
Weihao Gao (University of Illinois at Urbana-Champaign)
Yu-Han Liu (Google)
Chong Wang (ByteDance Inc.)
Sewoong Oh (University of Washington)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Oral: Rate Distortion For Model Compression:From Theory To Practice »
Tue. Jun 11th 10:15 -- 10:20 PM Room Room 102
More from the Same Authors
-
2021 : Robust and Differentially Private Covariance Estimation »
Logan Gnanapragasam · Jonathan Hayase · Sewoong Oh -
2023 : Unleashing the Power of Randomization in Auditing Differentially Private ML »
Krishna Pillutla · Galen Andrew · Peter Kairouz · Hugh B McMahan · Alina Oprea · Sewoong Oh -
2023 : Can Public Large Language Models Help Private Cross-device Federated Learning? »
Boxin Wang · Yibo J. Zhang · Yuan Cao · Bo Li · Hugh B McMahan · Sewoong Oh · Zheng Xu · Manzil Zaheer -
2023 : Can Public Large Language Models Help Private Cross-device Federated Learning? »
Boxin Wang · Yibo J. Zhang · Yuan Cao · Bo Li · Hugh B McMahan · Sewoong Oh · Zheng Xu · Manzil Zaheer -
2023 Poster: Why Is Public Pretraining Necessary for Private Model Training? »
Arun Ganesh · Mahdi Haghifam · Milad Nasresfahani · Sewoong Oh · Thomas Steinke · Om Thakkar · Abhradeep Guha Thakurta · Lun Wang -
2023 Poster: CRISP: Curriculum based Sequential neural decoders for Polar code family »
S Ashwin Hebbar · Viraj Nadkarni · Ashok Vardhan Makkuva · Suma Bhat · Sewoong Oh · Pramod Viswanath -
2023 Poster: Private Federated Learning with Autotuned Compression »
Enayat Ullah · Christopher Choquette-Choo · Peter Kairouz · Sewoong Oh -
2019 Poster: Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms »
Ashok Vardhan Makkuva · Pramod Viswanath · Sreeram Kannan · Sewoong Oh -
2019 Oral: Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms »
Ashok Vardhan Makkuva · Pramod Viswanath · Sreeram Kannan · Sewoong Oh -
2017 Poster: Sequence Modeling via Segmentations »
Chong Wang · Yining Wang · Po-Sen Huang · Abdelrahman Mohammad · Dengyong Zhou · Li Deng -
2017 Talk: Sequence Modeling via Segmentations »
Chong Wang · Yining Wang · Po-Sen Huang · Abdelrahman Mohammad · Dengyong Zhou · Li Deng