Timezone: »
Many existing compression approaches have been focused and evaluated on convolutional neural networks (CNNs) where fully-connected layers contain the most parameters (e.g., LeNet and AlexNet). However, the current trend of pushing CNNs deeper with convolutions has created a pressing demand to achieve higher compression gains on CNNs where convolutions dominate the parameter amount (e.g., GoogLeNet, ResNet and Wide ResNet). Further, convolutional layers always account for most energy consumption in run time. To this end, this paper investigates the relatively less-explored direction of compressing convolutional layers in deep CNNs. We introduce a novel spectrally relaxed k -means regularization, that tends to approximately make hard assignments of convolutional layer weights to K learned cluster centers during re-training. Compression is then achieved through weight-sharing, by only recording K cluster centers and weight assignment indexes. Our proposed pipeline, termed Deep k -Means, has well-aligned goals between re-training and compression stages. We further propose an improved set of metrics to estimate energy consumption of CNN hardware implementations, whose estimation results are verified to be consistent with previously proposed energy estimation tool extrapolated from actual hardware measurements. We have evaluated Deep k -Means in compressing several CNN models in terms of both compression ratio and energy consumption reduction, observing promising results without incurring accuracy loss.
Author Information
Junru Wu (Texas A&M University)
Yue Wang (Rice University)
Zhenyu Wu (Texas A&M University)
Zhangyang Wang (Texas A&M University)
Ashok Veeraraghavan (Rice University)
Yingyan Lin (Rice University)
Related Events (a corresponding poster, oral, or spotlight)
-
2018 Oral: Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions »
Fri Jul 13th 03:40 -- 03:50 PM Room Victoria
More from the Same Authors
-
2020 Poster: AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks »
Yonggan Fu · Wuyang Chen · Haotao Wang · Haoran Li · Yingyan Lin · Zhangyang Wang -
2019 Workshop: Joint Workshop on On-Device Machine Learning & Compact Deep Neural Network Representations (ODML-CDNNR) »
Sujith Ravi · Zornitsa Kozareva · Lixin Fan · Max Welling · Yurong Chen · Werner Bailer · Brian Kulis · Haoji Hu · Jonathan Dekhtiar · Yingyan Lin · Diana Marculescu -
2019 Poster: Plug-and-Play Methods Provably Converge with Properly Trained Denoisers »
Ernest Ryu · Jialin Liu · Sicheng Wang · Xiaohan Chen · Zhangyang Wang · Wotao Yin -
2019 Oral: Plug-and-Play Methods Provably Converge with Properly Trained Denoisers »
Ernest Ryu · Jialin Liu · Sicheng Wang · Xiaohan Chen · Zhangyang Wang · Wotao Yin -
2018 Poster: prDeep: Robust Phase Retrieval with a Flexible Deep Network »
Christopher Metzler · Phillip Schniter · Ashok Veeraraghavan · Richard Baraniuk -
2018 Oral: prDeep: Robust Phase Retrieval with a Flexible Deep Network »
Christopher Metzler · Phillip Schniter · Ashok Veeraraghavan · Richard Baraniuk