SimpleSpot and Evaluating Systemic Errors using Synthetic Image Datasets

We introduce SynthSpot, a framework for generating synthetic datasets to use for evaluating methods for discovering blindspots (i.e., systemic errors) in image classifiers, and SimpleSpot, a method for discovering such blindspots. We use SynthSpot to run controlled studies of how various factors influence blindspot discovery method performance. Our experimental results reveal several important shortcomings of existing methods, such as their relatively poor performance in settings with multiple model blindspots and their sensitivity to hyper-parameters. Further, we find that SimpleSpot is competitive with existing methods, which has promising implications for developing an interactive tool based on it.