Timezone: »

 
Active Learning under Pool Set Distribution Shift and Noisy Data
Yarin Gal · Tom Rainforth · Andreas Kirsch

Sat Jul 24 12:57 PM -- 01:02 PM (PDT) @
in Subset Selection in Machine Learning: From Theory to Applications »

Bayesian Active Learning has focused on BALD, which reduces model parameter uncertainty. However, we show that BALD gets stuck on out-of-distribution or junk data that is not relevant for the task. We examine a novel Expected Predictive Information Gain (EPIG) to deal with distribution shifts of the pool set. EPIG reduces the uncertainty of predictions on an unlabelled evaluation set sampled from the test data distribution whose distribution might be different to the pool set distribution. Based on this, our new EPIG-BALD acquisition function for Bayesian Neural Networks selects samples to improve the performance on the test data distribution instead of selecting samples that reduce model uncertainty everywhere, including for out-of-distribution regions with low density in the test data distribution. Our method outperforms state-of-the-art Bayesian active learning methods on high-dimensional datasets and avoids out-of-distribution junk data in cases where current state-of-the-art methods fail.

Author Information

Yarin Gal (University of Oxford)
Tom Rainforth (University of Oxford)
Andreas Kirsch (University of Oxford)

More from the Same Authors