Timezone: »

 
Poster
Beyond In-Domain Scenarios: Robust Density-Aware Calibration
Christian Tomani · Futa Waseda · Yuesong Shen · Daniel Cremers

Thu Jul 27 04:30 PM -- 06:00 PM (PDT) @ Exhibit Hall 1 #504
in Poster Session 6 »

Calibrating deep learning models to yield uncertainty-aware predictions is crucial as deep neural networks get increasingly deployed in safety-critical applications. While existing post-hoc calibration methods achieve impressive results on in-domain test datasets, they are limited by their inability to yield reliable uncertainty estimates in domain-shift and out-of-domain (OOD) scenarios. We aim to bridge this gap by proposing DAC, an accuracy-preserving as well as Density-Aware Calibration method based on k-nearest-neighbors (KNN). In contrast to existing post-hoc methods, we utilize hidden layers of classifiers as a source for uncertainty-related information and study their importance. We show that DAC is a generic method that can readily be combined with state-of-the-art post-hoc methods. DAC boosts the robustness of calibration performance in domain-shift and OOD, while maintaining excellent in-domain predictive uncertainty estimates. We demonstrate that DAC leads to consistently better calibration across a large number of model architectures, datasets, and metrics. Additionally, we show that DAC improves calibration substantially on recent large-scale neural networks pre-trained on vast amounts of data.

[ PDF]  [ Poster] 

Author Information

Christian Tomani (Technical University Munich)
Futa Waseda (The University of Tokyo)
Yuesong Shen (Technical University Munich)
Daniel Cremers (TU Munich)

More from the Same Authors