Timezone: »

 
SimpleSpot and Evaluating Systemic Errors using Synthetic Image Datasets
Gregory Plumb · Nari Johnson · Ángel Alexander Cabrera · Marco Ribeiro · Ameet Talwalkar

@
in Spurious correlations, Invariance, and Stability (SCIS) »
Event URL: https://openreview.net/forum?id=-NMPXRQlfc4 »

We introduce SynthSpot, a framework for generating synthetic datasets to use for evaluating methods for discovering blindspots (i.e., systemic errors) in image classifiers, and SimpleSpot, a method for discovering such blindspots. We use SynthSpot to run controlled studies of how various factors influence blindspot discovery method performance. Our experimental results reveal several important shortcomings of existing methods, such as their relatively poor performance in settings with multiple model blindspots and their sensitivity to hyper-parameters. Further, we find that SimpleSpot is competitive with existing methods, which has promising implications for developing an interactive tool based on it.

Author Information

Gregory Plumb (Carnegie Mellon University)
Nari Johnson (Carnegie Mellon University)
Ángel Alexander Cabrera (Carnegie Mellon University)
Marco Ribeiro
Ameet Talwalkar (Carnegie Mellon University)

More from the Same Authors