Timezone: »

Forgetting-free Continual Learning with Winning Subnetworks
Haeyong Kang · Rusty Mina · Sultan Rizky Hikmawan Madjid · Jaehong Yoon · Chang Yoo · Sung Ju Hwang · Mark Hasegawa-Johnson

Tue Jul 19 02:20 PM -- 02:25 PM (PDT) @ None

Inspired by Lottery Ticket Hypothesis that there exist competitive subnetworks within a full network, a continual learning method referred to as Winning Subnetworks (WSN) that sequentially learns and selects a subnetwork for each task is explored. More specifically, WSN jointly learns the model weights and task-adaptive binary masks pertaining to the subnetworks associated with each task whilst selecting a few weights to be activated (winning ticket) reusing the weight of the prior subnetworks. The proposed method is inherently immune to catastrophic forgetting as each selected subnetwork model does not infringe upon other selected subnetworks. The binary masks spawned per winning ticket are encoded into one N-bit binary digit mask which is then compressed using Huffman coding for a sub-linear increase in network capacity with respect to the number of tasks.

Author Information

Haeyong Kang (KAIST)
Rusty Mina (N/A)
Sultan Rizky Hikmawan Madjid (Korea Advanced Institute of Science and Technology)
Jaehong Yoon (KAIST)
Chang Yoo (KAIST)
Sung Ju Hwang (KAIST, AITRICS)
Mark Hasegawa-Johnson (University of Illinois)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors