What is Missing? Explaining Neurons Activated by Absent Concepts

Explainable artificial intelligence (XAI) aims to provide human-interpretable insights into the behavior of deep neural networks (DNNs), typically by estimating a simplified causal structure of the model. In existing work, this causal structure often includes relationships where the presence of a concept is associated with a strong activation of a neuron. For example, attribution methods primarily identify input pixels that contribute most to a prediction, and feature visualization methods reveal inputs that cause high activation of a target neuron – the former implicitly assuming that the relevant information resides in the input, and the latter that neurons encode the presence of concepts. However, a largely overlooked type of causal relationship is that of encoded absences, where the absence of a concept increases neural activation. In this work, we show that such missing but relevant concepts are common and that mainstream XAI methods struggle to reveal them when applied in their standard form. To address this, we propose two simple extensions to attribution and feature visualization techniques that uncover encoded absences. Across experiments, we show how mainstream XAI methods can be used to reveal and explain encoded absences, how ImageNet models exploit them, and that debiasing can be improved when considering them.

When trying to understand why an AI system made a certain decision, we usually look for the things it detected in the input. For example, if an image classifier predicts a certain dog breed, explanation methods might highlight concepts like the ears, fur, or snout. But models can also use information that is missing in the image, and current explanation methods mostly overlook this. In our work, we study these "missing" concepts. We show that neurons in neural networks can activate more strongly because some other concept is absent. For instance, a model predicting "Irish Setter" may rely on not seeing concepts that are typical for other dog breeds, such as humans would do. We show that this behavior is common in image classification models and that standard explanation methods often fail to reveal it. To address this, we adapt existing attribution and feature visualization methods so that they can also explain these missing concepts. We further show that these missing concepts can contribute to biases in the model, and that taking them into account can improve debiasing methods. Overall, our results suggest that understanding neural networks requires not only looking at what a model detects, but also at what it implicitly rules out.