Timezone: »
In many machine learning applications, it is important for the model to provide confidence scores that accurately capture its prediction uncertainty. Although modern learning methods have achieved great success in predictive accuracy, generating calibrated confidence scores remains a major challenge. Mixup, a popular yet simple data augmentation technique based on taking convex combinations of pairs of training examples, has been empirically found to significantly improve confidence calibration across diverse applications. However, when and how Mixup helps calibration is still a mystery. In this paper, we theoretically prove that Mixup improves calibration in \textit{high-dimensional} settings by investigating natural statistical models. Interestingly, the calibration benefit of Mixup increases as the model capacity increases. We support our theories with experiments on common architectures and datasets. In addition, we study how Mixup improves calibration in semi-supervised learning. While incorporating unlabeled data can sometimes make the model less calibrated, adding Mixup training mitigates this issue and provably improves calibration. Our analysis provides new insights and a framework to understand Mixup and calibration.
Author Information
Linjun Zhang (Rutgers University)
Zhun Deng (Harvard)
Kenji Kawaguchi (National University of Singapore)
James Zou (Stanford University)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: When and How Mixup Improves Calibration »
Wed. Jul 20th through Thu the 21st Room Hall E #523
More from the Same Authors
-
2021 : Meaningfully Explaining a Model's Mistakes »
· Abubakar Abid · James Zou -
2021 : Meaningfully Explaining a Model's Mistakes »
Abubakar Abid · James Zou -
2021 : MetaDataset: A Dataset of Datasets for Evaluating Distribution Shifts and Training Conflicts »
Weixin Liang · James Zou · Weixin Liang -
2021 : Have the Cake and Eat It Too? Higher Accuracy and Less Expense when Using Multi-label ML APIs Online »
Lingjiao Chen · James Zou · Matei Zaharia -
2021 : Machine Learning API Shift Assessments: Change is Coming! »
Lingjiao Chen · James Zou · Matei Zaharia -
2021 : Do Humans Trust Advice More if it Comes from AI? An Analysis of Human-AI Interactions »
Kailas Vodrahalli · James Zou -
2022 : On the nonlinear correlation of ML performance across data subpopulations »
Weixin Liang · Yining Mao · Yongchan Kwon · Xinyu Yang · James Zou -
2023 Poster: Auxiliary Learning as an Asymmetric Bargaining Game »
Aviv Shamsian · Aviv Navon · Neta Glazer · Kenji Kawaguchi · Gal Chechik · Ethan Fetaya -
2023 Poster: GFlowOut: Dropout with Generative Flow Networks »
Dianbo Liu · Moksh Jain · Bonaventure F. P. Dossou · Qianli Shen · Salem Lahlou · Anirudh Goyal · Nikolay Malkin · Chris Emezue · Dinghuai Zhang · Nadhir Hassen · Xu Ji · Kenji Kawaguchi · Yoshua Bengio -
2023 Poster: Discrete Key-Value Bottleneck »
Frederik Träuble · Anirudh Goyal · Nasim Rahaman · Michael Mozer · Kenji Kawaguchi · Yoshua Bengio · Bernhard Schölkopf -
2023 Poster: Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximation »
Jeffrey Willette · Seanie Lee · Bruno Andreis · Kenji Kawaguchi · Juho Lee · Sung Ju Hwang -
2023 Poster: Discover and Cure: Concept-aware Mitigation of Spurious Correlation »
Yingxin Wu · Mert Yuksekgonul · Linjun Zhang · James Zou -
2023 Poster: How Does Information Bottleneck Help Deep Learning? »
Kenji Kawaguchi · Zhun Deng · Xu Ji · Jiaoyang Huang -
2022 : GSCLIP : A Framework for Explaining Distribution Shifts in Natural Language »
Zhiying Zhu · Weixin Liang · James Zou -
2022 : Evaluation of ML in Health/Science »
James Zou -
2022 : Data Sculpting: Interpretable Algorithm for End-to-End Cohort Selection »
Ruishan Liu · James Zou -
2022 : Data Budgeting for Machine Learning »
Weixin Liang · James Zou -
2022 Poster: Efficient Online ML API Selection for Multi-Label Classification Tasks »
Lingjiao Chen · Matei Zaharia · James Zou -
2022 Poster: Improving Out-of-Distribution Robustness via Selective Augmentation »
Huaxiu Yao · Yu Wang · Sai Li · Linjun Zhang · Weixin Liang · James Zou · Chelsea Finn -
2022 Spotlight: Efficient Online ML API Selection for Multi-Label Classification Tasks »
Lingjiao Chen · Matei Zaharia · James Zou -
2022 Spotlight: Improving Out-of-Distribution Robustness via Selective Augmentation »
Huaxiu Yao · Yu Wang · Sai Li · Linjun Zhang · Weixin Liang · James Zou · Chelsea Finn -
2022 Poster: Robustness Implies Generalization via Data-Dependent Generalization Bounds »
Kenji Kawaguchi · Zhun Deng · Kyle Luh · Jiaoyang Huang -
2022 Poster: Multi-Task Learning as a Bargaining Game »
Aviv Navon · Aviv Shamsian · Idan Achituve · Haggai Maron · Kenji Kawaguchi · Gal Chechik · Ethan Fetaya -
2022 Oral: Robustness Implies Generalization via Data-Dependent Generalization Bounds »
Kenji Kawaguchi · Zhun Deng · Kyle Luh · Jiaoyang Huang -
2022 Spotlight: Multi-Task Learning as a Bargaining Game »
Aviv Navon · Aviv Shamsian · Idan Achituve · Haggai Maron · Kenji Kawaguchi · Gal Chechik · Ethan Fetaya -
2021 Poster: Improving Generalization in Meta-learning via Task Augmentation »
Huaxiu Yao · Long-Kai Huang · Linjun Zhang · Ying WEI · Li Tian · James Zou · Junzhou Huang · Zhenhui (Jessie) Li -
2021 Spotlight: Improving Generalization in Meta-learning via Task Augmentation »
Huaxiu Yao · Long-Kai Huang · Linjun Zhang · Ying WEI · Li Tian · James Zou · Junzhou Huang · Zhenhui (Jessie) Li -
2021 Poster: Toward Better Generalization Bounds with Locally Elastic Stability »
Zhun Deng · Hangfeng He · Weijie Su -
2021 Spotlight: Toward Better Generalization Bounds with Locally Elastic Stability »
Zhun Deng · Hangfeng He · Weijie Su -
2021 Poster: How to Learn when Data Reacts to Your Model: Performative Gradient Descent »
Zachary Izzo · Lexing Ying · James Zou -
2021 Spotlight: How to Learn when Data Reacts to Your Model: Performative Gradient Descent »
Zachary Izzo · Lexing Ying · James Zou -
2021 Poster: Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth »
Keyulu Xu · Mozhi Zhang · Stefanie Jegelka · Kenji Kawaguchi -
2021 Spotlight: Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth »
Keyulu Xu · Mozhi Zhang · Stefanie Jegelka · Kenji Kawaguchi -
2020 Poster: A Distributional Framework For Data Valuation »
Amirata Ghorbani · Michael Kim · James Zou -
2020 Poster: Interpreting Robust Optimization via Adversarial Influence Functions »
Zhun Deng · Cynthia Dwork · Jialiang Wang · Linjun Zhang -
2020 Poster: Towards Understanding the Dynamics of the First-Order Adversaries »
Zhun Deng · Hangfeng He · Jiaoyang Huang · Weijie Su -
2019 Poster: Concrete Autoencoders: Differentiable Feature Selection and Reconstruction »
Muhammed Fatih Balın · Abubakar Abid · James Zou -
2019 Poster: Discovering Conditionally Salient Features with Statistical Guarantees »
Jaime Roquero Gimenez · James Zou -
2019 Oral: Discovering Conditionally Salient Features with Statistical Guarantees »
Jaime Roquero Gimenez · James Zou -
2019 Oral: Concrete Autoencoders: Differentiable Feature Selection and Reconstruction »
Muhammed Fatih Balın · Abubakar Abid · James Zou -
2019 Poster: Data Shapley: Equitable Valuation of Data for Machine Learning »
Amirata Ghorbani · James Zou -
2019 Oral: Data Shapley: Equitable Valuation of Data for Machine Learning »
Amirata Ghorbani · James Zou -
2018 Poster: CoVeR: Learning Covariate-Specific Vector Representations with Tensor Decompositions »
Kevin Tian · Teng Zhang · James Zou -
2018 Oral: CoVeR: Learning Covariate-Specific Vector Representations with Tensor Decompositions »
Kevin Tian · Teng Zhang · James Zou