Timezone: »
Neural networks can be compressed to reduce memory and computational requirements, or to increase accuracy by facilitating the use of a larger base architecture. In this paper we focus on pruning individual neurons, which can simultaneously trim model size, FLOPs, and run-time memory. To improve upon the performance of existing compression algorithms we utilize the information bottleneck principle instantiated via a tractable variational bound. Minimization of this information theoretic bound reduces the redundancy between adjacent layers by aggregating useful information into a subset of neurons that can be preserved. In contrast, the activations of disposable neurons are shut off via an attractive form of sparse regularization that emerges naturally from this framework, providing tangible advantages over traditional sparsity penalties without contributing additional tuning parameters to the energy landscape. We demonstrate state-of-the-art compression rates across an array of datasets and network architectures.
Author Information
Bin Dai (Tsinghua University)
Chen Zhu (University of Maryland)
Baining Guo (MSR Asia)
David Wipf (Microsoft Research)
Related Events (a corresponding poster, oral, or spotlight)
-
2018 Poster: Compressing Neural Networks using the Variational Information Bottelneck »
Thu. Jul 12th 04:15 -- 07:00 PM Room Hall B #128
More from the Same Authors
-
2022 : Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Prior »
Ravid Shwartz-Ziv · Micah Goldblum · Hossein Souri · Sanyam Kapoor · Chen Zhu · Yann LeCun · Andrew Wilson -
2023 Poster: On the Initialization of Graph Neural Networks »
Jiahang Li · Yakun Song · Xiang song · David Wipf -
2023 Poster: From Hypergraph Energy Functions to Hypergraph Neural Networks »
Yuxin Wang · Quan Gan · Xipeng Qiu · Xuanjing Huang · David Wipf -
2023 Poster: Marginalization is not Marginal: No Bad VAE Local Minima when Learning Optimal Sparse Representations »
David Wipf -
2023 Oral: Marginalization is not Marginal: No Bad VAE Local Minima when Learning Optimal Sparse Representations »
David Wipf -
2022 : Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Prior »
Ravid Shwartz-Ziv · Micah Goldblum · Hossein Souri · Sanyam Kapoor · Chen Zhu · Yann LeCun · Andrew Wilson -
2022 Poster: Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations »
Amin Ghiasi · Hamid Kazemi · Steven Reich · Chen Zhu · Micah Goldblum · Tom Goldstein -
2022 Spotlight: Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentations »
Amin Ghiasi · Hamid Kazemi · Steven Reich · Chen Zhu · Micah Goldblum · Tom Goldstein -
2022 Poster: GNNRank: Learning Global Rankings from Pairwise Comparisons via Directed Graph Neural Networks »
Yixuan He · Quan Gan · David Wipf · Gesine Reinert · Junchi Yan · Mihai Cucuringu -
2022 Spotlight: GNNRank: Learning Global Rankings from Pairwise Comparisons via Directed Graph Neural Networks »
Yixuan He · Quan Gan · David Wipf · Gesine Reinert · Junchi Yan · Mihai Cucuringu -
2021 Poster: Graph Neural Networks Inspired by Classical Iterative Algorithms »
Yongyi Yang · Tang Liu · Yangkun Wang · Jinjing Zhou · Quan Gan · Zhewei Wei · Zheng Zhang · Zengfeng Huang · David Wipf -
2021 Oral: Graph Neural Networks Inspired by Classical Iterative Algorithms »
Yongyi Yang · Tang Liu · Yangkun Wang · Jinjing Zhou · Quan Gan · Zhewei Wei · Zheng Zhang · Zengfeng Huang · David Wipf -
2020 Poster: The Usual Suspects? Reassessing Blame for VAE Posterior Collapse »
Bin Dai · Ziyu Wang · David Wipf -
2019 Poster: Transferable Clean-Label Poisoning Attacks on Deep Neural Nets »
Chen Zhu · W. Ronny Huang · Hengduo Li · Gavin Taylor · Christoph Studer · Tom Goldstein -
2019 Oral: Transferable Clean-Label Poisoning Attacks on Deep Neural Nets »
Chen Zhu · W. Ronny Huang · Hengduo Li · Gavin Taylor · Christoph Studer · Tom Goldstein