Timezone: »
Most neural network pruning methods, such as filter-level and layer-level prunings, prune the network model along one dimension (depth, width, or resolution) solely to meet a computational budget. However, such a pruning policy often leads to excessive reduction of that dimension, thus inducing a huge accuracy loss. To alleviate this issue, we argue that pruning should be conducted along three dimensions comprehensively. For this purpose, our pruning framework formulates pruning as an optimization problem. Specifically, it first casts the relationships between a certain model's accuracy and depth/width/resolution into a polynomial regression and then maximizes the polynomial to acquire the optimal values for the three dimensions. Finally, the model is pruned along the three optimal dimensions accordingly. In this framework, since collecting too much data for training the regression is very time-costly, we propose two approaches to lower the cost: 1) specializing the polynomial to ensure an accurate regression even with less training data; 2) employing iterative pruning and fine-tuning to collect the data faster. Extensive experiments show that our proposed algorithm surpasses state-of-the-art pruning algorithms and even neural architecture search-based algorithms.
Author Information
Wenxiao Wang (ZJU)
Minghao Chen (Zhejiang University)
Shuai Zhao (The Chinese University of HongKong, Shenzhen)
Long Chen (Columbia University)
Jinming Hu (Microsoft)
Haifeng Liu (ZJU)
Deng Cai (ZJU)
Xiaofei He (Zhejiang University)
Wei Liu (Tencent AI Lab)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Poster: Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework »
Wed. Jul 21st 04:00 -- 06:00 AM Room
More from the Same Authors
-
2022 : Towards Multi-level Fairness and Robustness on Federated Learning »
Fengda Zhang · Kun Kuang · Yuxuan Liu · Long Chen · Jiaxun Lu · Yunfeng Shao · Fei Wu · Chao Wu · Jun Xiao -
2022 Poster: DynaMixer: A Vision MLP Architecture with Dynamic Mixing »
Ziyu Wang · Wenhao Jiang · Yiming Zhu · Li Yuan · Yibing Song · Wei Liu -
2022 Poster: Deconfounded Value Decomposition for Multi-Agent Reinforcement Learning »
Jiahui Li · Kun Kuang · Baoxiang Wang · Furui Liu · Long Chen · Changjie Fan · Fei Wu · Jun Xiao -
2022 Spotlight: Deconfounded Value Decomposition for Multi-Agent Reinforcement Learning »
Jiahui Li · Kun Kuang · Baoxiang Wang · Furui Liu · Long Chen · Changjie Fan · Fei Wu · Jun Xiao -
2022 Spotlight: DynaMixer: A Vision MLP Architecture with Dynamic Mixing »
Ziyu Wang · Wenhao Jiang · Yiming Zhu · Li Yuan · Yibing Song · Wei Liu -
2021 Poster: KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation »
Haozhe Feng · Zhaoyang You · Minghao Chen · Tianye Zhang · Minfeng Zhu · Fei Wu · Chao Wu · Wei Chen -
2021 Poster: CRFL: Certifiably Robust Federated Learning against Backdoor Attacks »
Chulin Xie · Minghao Chen · Pin-Yu Chen · Bo Li -
2021 Spotlight: CRFL: Certifiably Robust Federated Learning against Backdoor Attacks »
Chulin Xie · Minghao Chen · Pin-Yu Chen · Bo Li -
2021 Spotlight: KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation »
Haozhe Feng · Zhaoyang You · Minghao Chen · Tianye Zhang · Minfeng Zhu · Fei Wu · Chao Wu · Wei Chen -
2021 Poster: LARNet: Lie Algebra Residual Network for Face Recognition »
Xiaolong Yang · Xiaohong Jia · Dihong Gong · Dong-Ming Yan · Zhifeng Li · Wei Liu -
2021 Spotlight: LARNet: Lie Algebra Residual Network for Face Recognition »
Xiaolong Yang · Xiaohong Jia · Dihong Gong · Dong-Ming Yan · Zhifeng Li · Wei Liu -
2020 Poster: Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks »
Zhishuai Guo · Mingrui Liu · Zhuoning Yuan · Li Shen · Wei Liu · Tianbao Yang -
2020 Poster: Adversarial Mutual Information for Text Generation »
Boyuan Pan · Yazheng Yang · Kaizhao Liang · Bhavya Kailkhura · Zhongming Jin · Xian-Sheng Hua · Deng Cai · Bo Li -
2018 Poster: An Algorithmic Framework of Variable Metric Over-Relaxed Hybrid Proximal Extra-Gradient Method »
Li Shen · Peng Sun · Yitong Wang · Wei Liu · Tong Zhang -
2018 Oral: An Algorithmic Framework of Variable Metric Over-Relaxed Hybrid Proximal Extra-Gradient Method »
Li Shen · Peng Sun · Yitong Wang · Wei Liu · Tong Zhang -
2018 Poster: Safe Element Screening for Submodular Function Minimization »
Weizhong Zhang · Bin Hong · Lin Ma · Wei Liu · Tong Zhang -
2018 Poster: End-to-end Active Object Tracking via Reinforcement Learning »
Wenhan Luo · Peng Sun · Fangwei Zhong · Wei Liu · Tong Zhang · Yizhou Wang -
2018 Oral: End-to-end Active Object Tracking via Reinforcement Learning »
Wenhan Luo · Peng Sun · Fangwei Zhong · Wei Liu · Tong Zhang · Yizhou Wang -
2018 Oral: Safe Element Screening for Submodular Function Minimization »
Weizhong Zhang · Bin Hong · Lin Ma · Wei Liu · Tong Zhang