Finding Spuriously Correlated Visual Attributes

Deep neural models learn to use spurious features present in image datasets which hurts their out-of-distribution performance and makes them unreliable for critical application like medical imaging. To help develop robust models, it becomes essential to find spurious features in training datasets. Existing methods to find spurious features do not give any semantic meaning to the features and rely on human interpretation of the discovered correlated features to find if they are spurious or not. In this paper, we propose to first rotate the latent features into visual attributes and then learn correlation between the attributes and object classes by training a simple linear classifier. Correlated visual attributes are easily interpretable because they have well defined semantic meaning and makes it easier to find if they are spurious or not. Through visualizaions and experiments, we show how to find spurious visual attributes, their extent in existing dataset and failure mode examples showing negative impact of learned spurious correlations on out-of-distribution generalization.