Timezone: »
Many existing compression approaches have been focused and evaluated on convolutional neural networks (CNNs) where fully-connected layers contain the most parameters (e.g., LeNet and AlexNet). However, the current trend of pushing CNNs deeper with convolutions has created a pressing demand to achieve higher compression gains on CNNs where convolutions dominate the parameter amount (e.g., GoogLeNet, ResNet and Wide ResNet). Further, convolutional layers always account for most energy consumption in run time. To this end, this paper investigates the relatively less-explored direction of compressing convolutional layers in deep CNNs. We introduce a novel spectrally relaxed k -means regularization, that tends to approximately make hard assignments of convolutional layer weights to K learned cluster centers during re-training. Compression is then achieved through weight-sharing, by only recording K cluster centers and weight assignment indexes. Our proposed pipeline, termed Deep k -Means, has well-aligned goals between re-training and compression stages. We further propose an improved set of metrics to estimate energy consumption of CNN hardware implementations, whose estimation results are verified to be consistent with previously proposed energy estimation tool extrapolated from actual hardware measurements. We have evaluated Deep k -Means in compressing several CNN models in terms of both compression ratio and energy consumption reduction, observing promising results without incurring accuracy loss.
Author Information
Junru Wu (Texas A&M University)
Yue Wang (Rice University)
Zhenyu Wu (Texas A&M University)
Zhangyang Wang (Texas A&M University)
Ashok Veeraraghavan (Rice University)
Yingyan Lin (Rice University)
Related Events (a corresponding poster, oral, or spotlight)
-
2018 Oral: Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions »
Fri. Jul 13th 03:40 -- 03:50 PM Room Victoria
More from the Same Authors
-
2022 Poster: ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks »
Haoran You · Baopu Li · Shi Huihong · Yonggan Fu · Yingyan Lin -
2022 Poster: DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks »
Yonggan Fu · Haichuan Yang · Jiayi Yuan · Meng Li · Cheng Wan · Raghuraman Krishnamoorthi · Vikas Chandra · Yingyan Lin -
2022 Spotlight: DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks »
Yonggan Fu · Haichuan Yang · Jiayi Yuan · Meng Li · Cheng Wan · Raghuraman Krishnamoorthi · Vikas Chandra · Yingyan Lin -
2022 Spotlight: ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks »
Haoran You · Baopu Li · Shi Huihong · Yonggan Fu · Yingyan Lin -
2021 Poster: Double-Win Quant: Aggressively Winning Robustness of Quantized Deep Neural Networks via Random Precision Training and Inference »
Yonggan Fu · Qixuan Yu · Meng Li · Vikas Chandra · Yingyan Lin -
2021 Spotlight: Double-Win Quant: Aggressively Winning Robustness of Quantized Deep Neural Networks via Random Precision Training and Inference »
Yonggan Fu · Qixuan Yu · Meng Li · Vikas Chandra · Yingyan Lin -
2021 Poster: Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators »
Yonggan Fu · Yongan Zhang · Yang Zhang · David Cox · Yingyan Lin -
2021 Spotlight: Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators »
Yonggan Fu · Yongan Zhang · Yang Zhang · David Cox · Yingyan Lin -
2020 Poster: AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks »
Yonggan Fu · Wuyang Chen · Haotao Wang · Haoran Li · Yingyan Lin · Zhangyang “Atlas” Wang -
2019 : Spotlight »
Tyler Scott · Kiran Thekumparampil · Jonathan Aigrain · Rene Bidart · Priyadarshini Panda · Dian Ang Yap · Yaniv Yacoby · Raphael Gontijo Lopes · Alberto Marchisio · Erik Englesson · Wanqian Yang · Moritz Graule · Yi Sun · Daniel Kang · Mike Dusenberry · Min Du · Hartmut Maennel · Kunal Menda · Vineet Edupuganti · Luke Metz · David Stutz · Vignesh Srinivasan · Timo Sämann · Vineeth N Balasubramanian · Sina Mohseni · Rob Cornish · Judith Butepage · Zhangyang Wang · Bai Li · Bo Han · Honglin Li · Maksym Andriushchenko · Lukas Ruff · Meet P. Vadera · Yaniv Ovadia · Sunil Thulasidasan · Disi Ji · Gang Niu · Saeed Mahloujifar · Aviral Kumar · SANGHYUK CHUN · Dong Yin · Joyce Xu Xu · Hugo Gomes · Raanan Rohekar -
2019 Workshop: Joint Workshop on On-Device Machine Learning & Compact Deep Neural Network Representations (ODML-CDNNR) »
Sujith Ravi · Zornitsa Kozareva · Lixin Fan · Max Welling · Yurong Chen · Werner Bailer · Brian Kulis · Haoji Hu · Jonathan Dekhtiar · Yingyan Lin · Diana Marculescu -
2019 Poster: Plug-and-Play Methods Provably Converge with Properly Trained Denoisers »
Ernest Ryu · Jialin Liu · Sicheng Wang · Xiaohan Chen · Zhangyang Wang · Wotao Yin -
2019 Oral: Plug-and-Play Methods Provably Converge with Properly Trained Denoisers »
Ernest Ryu · Jialin Liu · Sicheng Wang · Xiaohan Chen · Zhangyang Wang · Wotao Yin -
2018 Poster: prDeep: Robust Phase Retrieval with a Flexible Deep Network »
Christopher Metzler · Phillip Schniter · Ashok Veeraraghavan · Richard Baraniuk -
2018 Oral: prDeep: Robust Phase Retrieval with a Flexible Deep Network »
Christopher Metzler · Phillip Schniter · Ashok Veeraraghavan · Richard Baraniuk