Timezone: »
Detecting out-of-distribution inputs is critical for the safe deployment of machine learning models in the real world. However, neural networks are known to suffer from the overconfidence issue, where they produce abnormally high confidence for both in- and out-of-distribution inputs. In this work, we show that this issue can be mitigated through Logit Normalization (LogitNorm)---a simple fix to the cross-entropy loss---by enforcing a constant vector norm on the logits in training. Our method is motivated by the analysis that the norm of the logit keeps increasing during training, leading to overconfident output. Our key idea behind LogitNorm is thus to decouple the influence of output’s norm during network optimization. Trained with LogitNorm, neural networks produce highly distinguishable confidence scores between in- and out-of-distribution data. Extensive experiments demonstrate the superiority of LogitNorm, reducing the average FPR95 by up to 42.30% on common benchmarks.
Author Information
Hongxin Wei (Nanyang Technological University)
RENCHUNZI XIE (Nanyang Technological University)
Hao Cheng (Nanjing University)
LEI FENG (Nanyang Technological University)
Bo An (Nanyang Technological University)
Sharon Li (University of Wisconsin-Madison)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: Mitigating Neural Network Overconfidence with Logit Normalization »
Wed. Jul 20th through Thu the 21st Room Hall E #919
More from the Same Authors
-
2021 : Contingency-Aware Influence Maximization: A Reinforcement Learning Approach »
Haipeng Chen · Wei Qiu · Han-Ching Ou · Bo An · Milind Tambe -
2022 : Are Vision Transformers Robust to Spurious Correlations ? »
Soumya Suvra Ghosal · Yifei Ming · Sharon Li -
2023 Poster: Mitigating Memorization of Noisy Labels by Clipping the Model Prediction »
Hongxin Wei · HUIPING ZHUANG · RENCHUNZI XIE · Lei Feng · Gang Niu · Bo An · Sharon Li -
2023 Poster: When and How Does Known Class Help Discover Unknown Ones? Provable Understanding Through Spectral Analysis »
Yiyou Sun · Zhenmei Shi · Yingyiu Liang · Sharon Li -
2023 Poster: Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detection »
Haoyue Bai · Gregory Canal · Xuefeng Du · Jeongyeol Kwon · Robert Nowak · Sharon Li -
2022 Workshop: Workshop on Distribution-Free Uncertainty Quantification »
Anastasios Angelopoulos · Stephen Bates · Sharon Li · Ryan Tibshirani · Aaditya Ramdas · Stephen Bates -
2022 : Challenges and Opportunities in Handling Data Distributional Shift »
Sharon Li -
2022 Poster: Out-of-Distribution Detection with Deep Nearest Neighbors »
Yiyou Sun · Yifei Ming · Jerry Zhu · Sharon Li -
2022 Poster: Training OOD Detectors in their Natural Habitats »
Julian Katz-Samuels · Julia Nakhleh · Robert Nowak · Sharon Li -
2022 Poster: Learning Pseudometric-based Action Representations for Offline Reinforcement Learning »
Pengjie Gu · Mengchen Zhao · Chen Chen · Dong Li · Jianye Hao · Bo An -
2022 Spotlight: Learning Pseudometric-based Action Representations for Offline Reinforcement Learning »
Pengjie Gu · Mengchen Zhao · Chen Chen · Dong Li · Jianye Hao · Bo An -
2022 Spotlight: Training OOD Detectors in their Natural Habitats »
Julian Katz-Samuels · Julia Nakhleh · Robert Nowak · Sharon Li -
2022 Spotlight: Out-of-Distribution Detection with Deep Nearest Neighbors »
Yiyou Sun · Yifei Ming · Jerry Zhu · Sharon Li -
2022 Poster: POEM: Out-of-Distribution Detection with Posterior Sampling »
Yifei Ming · Ying Fan · Sharon Li -
2022 Poster: Open-Sampling: Exploring Out-of-Distribution data for Re-balancing Long-tailed datasets »
Hongxin Wei · Lue Tao · RENCHUNZI XIE · LEI FENG · Bo An -
2022 Oral: POEM: Out-of-Distribution Detection with Posterior Sampling »
Yifei Ming · Ying Fan · Sharon Li -
2022 Spotlight: Open-Sampling: Exploring Out-of-Distribution data for Re-balancing Long-tailed datasets »
Hongxin Wei · Lue Tao · RENCHUNZI XIE · LEI FENG · Bo An -
2021 : LOOD: Localization-based Uncertainty Estimation for Medical Imaging (Spotlight #14) »
Yiyou Sun · Sharon Li -
2021 Workshop: Workshop on Distribution-Free Uncertainty Quantification »
Anastasios Angelopoulos · Stephen Bates · Sharon Li · Aaditya Ramdas · Ryan Tibshirani -
2021 Workshop: Uncertainty and Robustness in Deep Learning »
Balaji Lakshminarayanan · Dan Hendrycks · Sharon Li · Jasper Snoek · Silvia Chiappa · Sebastian Nowozin · Thomas Dietterich -
2021 Poster: Pointwise Binary Classification with Pairwise Confidence Comparisons »
Lei Feng · Senlin Shu · Nan Lu · Bo Han · Miao Xu · Gang Niu · Bo An · Masashi Sugiyama -
2021 Poster: Learning from Similarity-Confidence Data »
Yuzhou Cao · Lei Feng · Yitian Xu · Bo An · Gang Niu · Masashi Sugiyama -
2021 Spotlight: Learning from Similarity-Confidence Data »
Yuzhou Cao · Lei Feng · Yitian Xu · Bo An · Gang Niu · Masashi Sugiyama -
2021 Spotlight: Pointwise Binary Classification with Pairwise Confidence Comparisons »
Lei Feng · Senlin Shu · Nan Lu · Bo Han · Miao Xu · Gang Niu · Bo An · Masashi Sugiyama -
2020 Poster: Learning Efficient Multi-agent Communication: An Information Bottleneck Approach »
Rundong Wang · Xu He · Runsheng Yu · Wei Qiu · Bo An · Zinovi Rabinovich -
2020 Poster: Progressive Identification of True Labels for Partial-Label Learning »
Jiaqi Lv · Miao Xu · LEI FENG · Gang Niu · Xin Geng · Masashi Sugiyama -
2020 Poster: Learning with Multiple Complementary Labels »
LEI FENG · Takuo Kaneko · Bo Han · Gang Niu · Bo An · Masashi Sugiyama