Timezone: »

 
Poster
Individual Calibration with Randomized Forecasting
Shengjia Zhao · Tengyu Ma · Stefano Ermon

Wed Jul 15 08:00 AM -- 08:45 AM & Wed Jul 15 07:00 PM -- 07:45 PM (PDT) @
in Poster Session 22 »

Machine learning applications often require calibrated predictions, e.g. a 90\% credible interval should contain the true outcome 90\% of the times. However, typical definitions of calibration only require this to hold on average, and offer no guarantees on predictions made on individual samples. Thus, predictions can be systematically over or under confident on certain subgroups, leading to issues of fairness and potential vulnerabilities. We show that calibration for individual samples is possible in the regression setup if and only if the predictions are randomized, i.e. outputting randomized credible intervals. Randomization removes systematic bias by trading off bias with variance. We design a training objective to enforce individual calibration and use it to train randomized regression functions. The resulting models are more calibrated for arbitrarily chosen subgroups of the data, and can achieve higher utility in decision making against adversaries that exploit miscalibrated predictions.

Keywords: Robust Statistics and Machine Learning · Statistical Learning Theory · Trustworthy Machine Learning · Fairness, Equity and Justice

Author Information

Shengjia Zhao (Stanford University)
Tengyu Ma (Stanford)
Stefano Ermon (Stanford University)

More from the Same Authors