Timezone: »

 
Poster
SIGUA: Forgetting May Make Learning with Noisy Labels More Robust
Bo Han · Gang Niu · Xingrui Yu · QUANMING YAO · Miao Xu · Ivor Tsang · Masashi Sugiyama

Wed Jul 15 05:00 AM -- 05:45 AM & Wed Jul 15 07:00 PM -- 07:45 PM (PDT) @
in Poster Session 21 »

Given data with noisy labels, over-parameterized deep networks can gradually memorize the data, and fit everything in the end. Although equipped with corrections for noisy labels, many learning methods in this area still suffer overfitting due to undesired memorization. In this paper, to relieve this issue, we propose stochastic integrated gradient underweighted ascent (SIGUA): in a mini-batch, we adopt gradient descent on good data as usual, and learning-rate-reduced gradient ascent} on bad data; the proposal is a versatile approach where data goodness or badness is w.r.t. desired or undesired memorization given a base learning method. Technically, SIGUA pulls optimization back for generalization when their goals conflict with each other; philosophically, SIGUA shows forgetting undesired memorization can reinforce desired memorization. Experiments demonstrate that SIGUA successfully robustifies two typical base learning methods, so that their performance is often significantly improved.

Keywords: Algorithms · Semi-supervised learning · Deep Learning - Algorithms

Author Information

Bo Han (HKBU / RIKEN)
Gang Niu (RIKEN)
Gang Niu

Gang Niu is currently an indefinite-term senior research scientist at RIKEN Center for Advanced Intelligence Project.

Xingrui Yu (University of Technology Sydney)
QUANMING YAO (4Paradigm)

Dr. Quanming Yao is currently a leading researcher in 4Paradigm and managing the company's research group. He obtained his Ph.D. degree at the Department of Computer Science and Engineering at Hong Kong University of Science and Technology (HKUST) in 2018 and received his bachelor degree at HuaZhong University of Science and Technology (HUST) in 2013. He is Qiming Star (HUST, 2012), Tse Cheuk Ng Tai Research Excellence Prize (CSE, HKUST, 2014-2015), Google Fellowship (machine learning, 2016) and Ph.D. Research Excellence Award (School of Engineering, HKUST, 2018-2019). He has 23 top-tier journal and conference papers, including ICML, NeurIPS, JMLR, TPAMI, KDD, ICDE, CVPR, IJCAI, and AAAI; he was an outstanding reviewer of Neurocomputing in 2017; served as program committee of many prestigious conferences, including ICML, NeurIPS, CVPR, AAAI, and IJCAI; one of the committees of AutoML competition in NeurIPS-2018, IJCNN-2019 and IJCAI-2019.

Miao Xu (University of Queensland/ RIKEN AIP)
Ivor Tsang (University of Technology Sydney)
Masashi Sugiyama (RIKEN / The University of Tokyo)

More from the Same Authors