Timezone: »

 
Spotlight
Forget-free Continual Learning with Winning Subnetworks
Haeyong Kang · Rusty Mina · Sultan Rizky Hikmawan Madjid · Jaehong Yoon · Mark Hasegawa-Johnson · Sung Ju Hwang · Chang Yoo

Tue Jul 19 02:20 PM -- 02:25 PM (PDT) @ Room 327 - 329
in DL: Sequential Models »

Inspired by Lottery Ticket Hypothesis that competitive subnetworks exist within a dense network, we propose a continual learning method referred to as Winning SubNetworks (WSN), which sequentially learns and selects an optimal subnetwork for each task. Specifically, WSN jointly learns the model weights and task-adaptive binary masks pertaining to subnetworks associated with each task whilst attempting to select a small set of weights to be activated (winning ticket) by reusing weights of the prior subnetworks. The proposed method is inherently immune to catastrophic forgetting as each selected subnetwork model does not infringe upon other subnetworks. Binary masks spawned per winning ticket are encoded into one N-bit binary digit mask, then compressed using Huffman coding for a sub-linear increase in network capacity with respect to the number of tasks.

Keywords: DL: Algorithms · DL: Attention Mechanisms · DL: Graph Neural Networks · DL: Robustness · DL: Self-Supervised Learning · DL: Sequential Models, Time series · MISC: Scalable Algorithms · MISC: Supervised Learning · MISC: Online Learning, Active Learning and Bandits · MISC: Sequential, Network, and Time Series Modeling · MISC: Transfer, Multitask and Meta-learning

Author Information

Haeyong Kang (KAIST)
Rusty Mina (N/A)
Sultan Rizky Hikmawan Madjid (Korea Advanced Institute of Science and Technology)
Jaehong Yoon (KAIST)
Mark Hasegawa-Johnson (University of Illinois)
Sung Ju Hwang (KAIST, AITRICS)
Chang Yoo (KAIST)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors