Timezone: »

 
Spotlight
Distribution-Free Calibration Guarantees for Histogram Binning without Sample Splitting
Chirag Gupta · Aaditya Ramdas

Wed Jul 21 06:45 AM -- 06:50 AM (PDT) @ None

We prove calibration guarantees for the popular histogram binning (also called uniform-mass binning) method of Zadrozny and Elkan (2001). Histogram binning has displayed strong practical performance, but theoretical guarantees have only been shown for sample split versions that avoid 'double dipping' the data. We demonstrate that the statistical cost of sample splitting is practically significant on a credit default dataset. We then prove calibration guarantees for the original method that double dips the data, using a certain Markov property of order statistics. Based on our results, we make practical recommendations for choosing the number of bins in histogram binning. In our illustrative simulations, we propose a new tool for assessing calibration---validity plots---which provide more information than an ECE estimate.

Author Information

Chirag Gupta (Carnegie Mellon University)
Aaditya Ramdas (Carnegie Mellon University)

Aaditya Ramdas is an assistant professor in the Departments of Statistics and Machine Learning at Carnegie Mellon University. These days, he has 3 major directions of research: 1. selective and simultaneous inference (interactive, structured, post-hoc control of false discovery/coverage rate,…), 2. sequential uncertainty quantification (confidence sequences, always-valid p-values, bias in bandits,…), and 3. assumption-free black-box predictive inference (conformal prediction, calibration,…).

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors