Timezone: »

 
Spotlight
When and How Mixup Improves Calibration
Linjun Zhang · Zhun Deng · Kenji Kawaguchi · James Zou

Wed Jul 20 07:40 AM -- 07:45 AM (PDT) @ Ballroom 3 & 4
in MISC: General Machine Learning Techniques »

In many machine learning applications, it is important for the model to provide confidence scores that accurately capture its prediction uncertainty. Although modern learning methods have achieved great success in predictive accuracy, generating calibrated confidence scores remains a major challenge. Mixup, a popular yet simple data augmentation technique based on taking convex combinations of pairs of training examples, has been empirically found to significantly improve confidence calibration across diverse applications. However, when and how Mixup helps calibration is still a mystery. In this paper, we theoretically prove that Mixup improves calibration in \textit{high-dimensional} settings by investigating natural statistical models. Interestingly, the calibration benefit of Mixup increases as the model capacity increases. We support our theories with experiments on common architectures and datasets. In addition, we study how Mixup improves calibration in semi-supervised learning. While incorporating unlabeled data can sometimes make the model less calibrated, adding Mixup training mitigates this issue and provably improves calibration. Our analysis provides new insights and a framework to understand Mixup and calibration.

Keywords: Theory · MISC: General Machine Learning Techniques · MISC: Supervised Learning · MISC: Unsupervised and Semi-supervised Learning · SA: Trustworthy Machine Learning

Author Information

Linjun Zhang (Rutgers University)
Zhun Deng (Harvard)
Kenji Kawaguchi (National University of Singapore)
James Zou (Stanford University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors