Tree-based machine learning models are popular nonlinear predictive models, yet comparatively little attention has been paid to explaining their predictions. In this talk I will explain how to improve their interpretability through the combination of many local game-theoretic explanations. I'll show how combining many high-quality local explanations allows us to represent global structure while retaining local faithfulness to the original model. This will enable us to identify high-magnitude but low-frequency nonlinear mortality risk factors in the US population, to highlight distinct population subgroups with shared risk characteristics, and to identify nonlinear interaction effects among risk factors for chronic kidney disease, and to monitor a machine learning model deployed in a hospital by identifying which features are degrading the model’s performance over time.

Recent progress in deep generative models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) has enabled synthesizing photo-realistic images, such as faces and scenes. However, it remains much less explored on what has been learned inside the deep representations learned from synthesizing images. In this talk, I will present some of our recent progress in interpreting the semantics in the latent space of the GANs, as well as reversing real images back into the latent space. Identifying these semantics not only allows us to better understand the internal mechanism in generative models, but also facilitates versatile real image editing applications.

This paper explains predictions of image captioning attention models beyond visualizing the attention itself. In this paper, we develop variants of layer-wise relevance backpropagation (LRP) tailored to image captioning models with attention mechanisms. We show that the explanations, firstly, correlate to object locations with higher precision than attention, secondly, identify object words that are unsupported by image content, and thirdly, provide guidance to improve the model. Results are reported using two different image captioning attention models trained with Flickr30K and MSCOCO2017 datasets. Experimental analyses show the strength of explanation methods for understanding image captioning attention models.

Recent work has discussed the limitations of counterfactual explanations to recommend actions for algorithmic recourse and argued for the need of taking causal relationships between features into consideration. Unfortunately, in practice, the true underlying structural causal model is generally unknown. In this work, we first show that it is impossible to guarantee recourse without access to the true structural equations. To address this limitation, we propose two probabilistic approaches to select optimal actions that achieve recourse with high probability given limited causal knowledge (e.g., only the causal graph). The first captures uncertainty over structural equations under additive Gaussian noise and uses Bayesian model averaging to estimate the counterfactual distribution. The second removes any assumptions on the structural equations by instead computing the average effect of recourse actions on individuals similar to the person who seeks recourse, leading to a novel subpopulation-based interventional notion of recourse. We derive a gradient-based procedure for selecting optimal recourse actions and empirically show that the proposed approaches lead to more reliable recommendations under imperfect causal knowledge than non-probabilistic baselines.

Zoom Room 1

• Sun et al. "Understanding Image Captioning Models beyond Visualizing Attention"
• Sun et al. "Explain and Improve: Cross-Domain-Few-Shot-Learning Using Explanations"
• Manupriya et al. "SEA-NN: Submodular Ensembled Attribution for Neural Networks"

[ protected link dropped ]

Zoom Room 2

• Macdonald et al. "Explaining Neural Network Decisions Is Hard"
• Molnar et al. "Pitfalls to Avoid when Interpreting Machine Learning Models"

[ protected link dropped ]

Unsupervised models such as clustering or anomaly detection are routinely used for data discovery and summarization. To gain maximum insight from the data, we also need to explain which input features (e.g. pixels) support the cluster assignments and the anomaly detections.—So far, XAI has mainly focused on supervised models.—In this talk, a novel systematic approach to explain various unsupervised models is presented. The approach is based on finding, without retraining, neural network equivalents of these models. Their predictions can then be readily explained using common XAI procedures developed for neural networks.

An agent who interacts with a wide population of other agents needs to be aware that there may be variations in their understanding of the world. Furthermore, the machinery which they use to perceive may be inherently different, as is the case between humans and machines. In this work, we present both an image reference game between a speaker and a population of listeners where reasoning about the concepts other agents can comprehend is necessary and a model formulation with this capability. We focus on reasoning about the conceptual understanding of others, as well as adapting to novel gameplay partners and dealing with differences in perceptual machinery. Our experiments on three benchmark image/attribute datasets suggest that our learner indeed encodes information directly pertaining to the understanding of other agents, and that leveraging this information is crucial for maximizing gameplay performance.

Assigning credit for a received reward to previously performed actions is one of the central tasks in reinforcement learning. Credit assignment often uses world models, either in a forward or in a backward view. In a forward view, the future return is estimated by replacing the environment through a model or by rolling out sequences until episode end. A backward view either learns a backward model or performs a backward analysis of a forward model that predicts or models the return of an episode. Our method RUDDER performs a backward analysis to construct a reward redistribution to credit those actions that caused a reward. Its extension Align-RUDDER learns a reward redistribution from few demonstrations. An optimal reward redistribution has zero expected future reward and, therefore, immediately credits actions for all they will cause. XAI aims at credit assignment, too, when asking what caused a model to produce a particular output given an input. Even further, XAI wants to know how and why a policy solved a task, why an agent is better than humans, why a decision was made. Humans best comprehend a strategy of an agent if all its actions are immediately evaluated and do not have hidden consequences in the future. Reward redistributions learned by RUDDER and Align-RUDDER help to understand task-solving strategies of both humans and machines.

For some time now, machine learning methods have been indispensable in many application areas. Especially with the recent development of neural networks, these methods are increasingly used in the sciences to obtain scientific outcomes from observational or simulated data. Besides a high accuracy, a desired goal is to learn explainable models. In order to reach this goal and obtain explanation, knowledge from the respective domain is necessary, which can be integrated into the model or applied post-hoc. This talk focuses on explainable machine learning approaches which are used to tackle common challenges in the sciences such as the provision of reliable and scientific consistent results. It will show that recent advances in machine learning to enhance transparency, interpretability, and explainability are helpful in overcoming these challenges.

Explainable machine learning offers the potential to provide stakeholders with insights into model behavior, yet there is little understanding of how organizations use these methods in practice. In this talk, we discuss recent research exploring how organizations view and use explainability. We find that the majority of deployments are not for end-users but rather for machine learning engineers, who use explainability to debug the model. There is thus a gap between explainability in practice and the goal of external transparency since explanations are primarily serving internal stakeholders. Providing useful external explanations requires careful consideration of the needs of stakeholders, including end-users, regulators, and domain experts. Despite this need, little work has been done to facilitate inter-stakeholder conversation around explainable machine learning. To help address this gap, we report findings from a closed-door, day-long workshop between academics, industry experts, legal scholars, and policymakers to develop a shared language around explainability and to understand the current shortcomings of and potential solutions for deploying explainable machine learning in the service of external transparency goals.

Structured control policies such as decision trees, finite-state machines, and programs have a number of advantages over more traditional models: they are easier for humans to understand and debug, they generalize more robustly to novel environments, and they are easier to formally verify. However, learning these kinds of models has proven to be challenging. I will describe recent progress learning structured policies, along with evidence demonstrating their benefits.

Contemporary learning models for computer vision are typically trained on very large data sets with millions of samples. There may, however, be biases, artifacts, or errors in the data that have gone unnoticed and are exploitable by the model, which in turn becomes a biased ‘Clever-Hans‘ predictor. In this paper, we contribute by providing a comprehensive analysis framework based on a scalable statistical analysis of attributions from explanation methods for large data corpora, here ImageNet. Based on Spectral Relevance Analysis we propose the following technical contributions and resulting findings: (a) a scalable quantification of artifactual classes where the ML models under study exhibit Clever-Hans behavior, (b) an approach denoted as Class-Artifact Compensation (ClArC) that allows to fine-tune an existing model to effectively eliminate its focus on artifacts and biases yielding significantly reduced Clever-Hans behavior.

We present a novel form of explanation for Reinforcement Learning (RL), based around the notion of intended outcome. This describes what outcome an agent is trying to achieve by its actions. Given this definition, we provide a simple proof that general methods for post-hoc explanations of this nature are impossible in traditional reinforcement learning. Rather, the information needed for the explanations must be collected in conjunction with training the agent. We provide approaches designed to do this for several variants of Q-function approximation and prove consistency between the explanations and the Q-values learned. We demonstrate our method on multiple reinforcement learning problems.

Zoom Room 1

• Bhatt et al. "Machine Learning Explainability for External Stakeholders"
• Karimi et al. "Algorithmic recourse under imperfect causal knowledge: a probabilistic approach"
• Wang et al. "Towards Probabilistic Sufficient Explanations"

[ protected link dropped ]

Zoom Room 2

• Agarwal et al. "Neural Additive Models: Interpretable Machine Learning with Neural Nets"
• Alaniz et al. "Learning Decision Trees Recurrently through Communication"
• Anders et al. "XAI for Analyzing and Unlearning Spurious Correlations in ImageNet"

[ protected link dropped ]

Zoom Room 3

• Dasgupta et al. "Explainable k-Means Clustering: Theory and Practice"
• Dhurandhar et al. "Leveraging Simple Model Predictions for Enhancing its Performance"
• Brophy and Lowd: "TREX: Tree-Ensemble Representer-Point Explanations"

[ protected link dropped ]

Zoom Room 4

• Lin et al. "Contrastive Explanations for Reinforcement Learning via Embedded Self Predictions"
• Yau et al. "What Did You Think Would Happen? Explaining Agent Behaviour through Intended Outcomes"
• Danesh et al. "Understanding Finite-State Representations of Recurrent Policy Networks"

[ protected link dropped ]

Zoom Room 5

• Quint et al. "Contrastive Attribution with Feature Visualization"
• Zhao "Fast Real-time Counterfactual Explanations"
• Chrysos et al. "Unsupervised Controllable Generation with Self-Training"

[ protected link dropped ]