Timezone: »

 
Keynote by Kilian Weinberger: On Calibration and Fairness
Kilian Weinberger

Fri Jun 14 11:00 AM -- 11:30 AM (PDT) @
in Uncertainty and Robustness in Deep Learning »

We investigate calibration for deep learning algorithms in classification and regression settings. Although we show that typically deep networks tend to be highly mis-calibrated, we demonstrate that this is easy to fix - either to obtain more trustworthy confidence estimates or to detect outliers in the data. Finally, we relate calibration with the recently raised tension between minimizing error disparity across different population groups while maintaining calibrated probability estimates. We show that calibration is compatible only with a single error constraint (i.e. equal false-negatives rates across groups), and show that any algorithm that satisfies this relaxation is no better than randomizing a percentage of predictions for an existing classifier. These unsettling findings, which extend and generalize existing results, are empirically confirmed on several datasets.

[ Video] 

Author Information

Kilian Weinberger (Cornell University)

Kilian Weinberger is an Associate Professor in the Department of Computer Science at Cornell University. He received his Ph.D. from the University of Pennsylvania in Machine Learning under the supervision of Lawrence Saul and his undergraduate degree in Mathematics and Computer Science from the University of Oxford. During his career he has won several best paper awards at ICML, CVPR, AISTATS and KDD (runner-up award). In 2011 he was awarded the Outstanding AAAI Senior Program Chair Award and in 2012 he received an NSF CAREER award. He was elected co-Program Chair for ICML 2016 and for AAAI 2018. Kilian Weinberger's research focuses on Machine Learning and its applications. In particular, he focuses on learning under resource constraints, metric learning, machine learned web-search ranking, computer vision and deep learning. Before joining Cornell University, he was an Associate Professor at Washington University in St. Louis and before that he worked as a research scientist at Yahoo! Research in Santa Clara.

More from the Same Authors