Timezone: »

 
Spotlight
Lottery Ticket Preserves Weight Correlation: Is It Desirable or Not?
Ning Liu · Geng Yuan · Zhengping Che · Xuan Shen · Xiaolong Ma · Qing Jin · Jian Ren · Jian Tang · Sijia Liu · Yanzhi Wang

Tue Jul 20 05:30 PM -- 05:35 PM (PDT) @
in Deep Learning 2 »

In deep model compression, the recent finding "Lottery Ticket Hypothesis" (LTH) pointed out that there could exist a winning ticket (i.e., a properly pruned sub-network together with original weight initialization) that can achieve competitive performance than the original dense network. However, it is not easy to observe such winning property in many scenarios, where for example, a relatively large learning rate is used even if it benefits training the original dense model. In this work, we investigate the underlying condition and rationale behind the winning property, and find that the underlying reason is largely attributed to the correlation between initialized weights and final-trained weights when the learning rate is not sufficiently large. Thus, the existence of winning property is correlated with an insufficient DNN pretraining, and is unlikely to occur for a well-trained DNN. To overcome this limitation, we propose the "pruning & fine-tuning" method that consistently outperforms lottery ticket sparse training under the same pruning algorithm and the same total training epochs. Extensive experiments over multiple deep models (VGG, ResNet, MobileNet-v2) on different datasets have been conducted to justify our proposals.

Keywords: Deep Learning

Author Information

Ning Liu (Midea Group)
Geng Yuan (Northeastern University)
Zhengping Che (Didi Chuxing)
Xuan Shen (Northeastern University)
Xiaolong Ma (Northeastern University)
Qing Jin (Northeastern University)
Jian Ren (Snap Inc.)
Jian Tang (AI Innovation Center, Midea Group)
Sijia Liu (Michigan State University)
Yanzhi Wang (Northeastern University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors