Timezone: »
Oral
Counterfactual Visual Explanations
Yash Goyal · Ziyan Wu · Jan Ernst · Dhruv Batra · Devi Parikh · Stefan Lee
Wed Jun 12 05:10 PM -- 05:15 PM (PDT) @ Seaside Ballroom
A counterfactual query is typically of the form ``For situation X, why was the outcome Y and not Z?''. A counterfactual explanation (or response to such a query) is of the form ``If X was X*, then the outcome would have been Z rather than Y.''
In this work, we develop a technique to produce counterfactual visual explanations. Given a `query' image $I$ for which a vision system predicts class $c$, a counterfactual visual explanation identifies how $I$ could change such that the system would output a different specified class $c'$. To do this, we select a `distractor' image $I'$ that the system predicts as class $c'$ and identify spatial regions in $I$ and $I'$ such that replacing the identified region in $I$ with the identified region in $I'$ would push the system towards classifying $I$ as $c'$.
We apply our approach to multiple image classification datasets generating qualitative results showcasing the intepretability and discriminativeness of our counterfactual explanations. To explore the effectiveness of our explanations in teaching humans, we present machine teaching experiments for the task of fine-grained bird classification. We find that users trained to distinguish bird species fare better when given access to counterfactual explanations in addition to training examples.
Author Information
Yash Goyal (Georgia Tech)
Ziyan Wu (Siemens Corporate Technology)
Jan Ernst (Siemens Corporation)
Dhruv Batra (Georgia Institute of Technology / Facebook AI Research)
Devi Parikh (Georgia Tech & Facebook AI Research)
Stefan Lee (Georgia Institute of Technology)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: Counterfactual Visual Explanations »
Thu. Jun 13th 01:30 -- 04:00 AM Room Pacific Ballroom #149
More from the Same Authors
-
2023 Poster: Text-To-4D Dynamic Scene Generation »
Uriel Singer · Shelly Sheynin · Adam Polyak · Oron Ashual · Iurii Makarov · Filippos Kokkinos · Naman Goyal · Andrea Vedaldi · Devi Parikh · Justin Johnson · Yaniv Taigman -
2023 Poster: Adaptive Coordination in Social Embodied Rearrangement »
Andrew Szot · Unnat Jain · Dhruv Batra · Zsolt Kira · Ruta Desai · Akshara Rai -
2019 : Forcing Vision + Language Models To Actually See, Not Just Talk »
Devi Parikh -
2019 Poster: Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering »
Shanmukha Ramakrishna Vedantam · Karan Desai · Stefan Lee · Marcus Rohrbach · Dhruv Batra · Devi Parikh -
2019 Poster: TarMAC: Targeted Multi-Agent Communication »
Abhishek Das · Theophile Gervet · Joshua Romoff · Dhruv Batra · Devi Parikh · Michael Rabbat · Joelle Pineau -
2019 Poster: Trainable Decoding of Sets of Sequences for Neural Sequence Models »
Ashwin Kalyan · Peter Anderson · Stefan Lee · Dhruv Batra -
2019 Oral: TarMAC: Targeted Multi-Agent Communication »
Abhishek Das · Theophile Gervet · Joshua Romoff · Dhruv Batra · Devi Parikh · Michael Rabbat · Joelle Pineau -
2019 Oral: Probabilistic Neural Symbolic Models for Interpretable Visual Question Answering »
Shanmukha Ramakrishna Vedantam · Karan Desai · Stefan Lee · Marcus Rohrbach · Dhruv Batra · Devi Parikh -
2019 Oral: Trainable Decoding of Sets of Sequences for Neural Sequence Models »
Ashwin Kalyan · Peter Anderson · Stefan Lee · Dhruv Batra -
2018 Poster: Learn from Your Neighbor: Learning Multi-modal Mappings from Sparse Annotations »
Ashwin Kalyan · Stefan Lee · Anitha Kannan · Dhruv Batra -
2018 Oral: Learn from Your Neighbor: Learning Multi-modal Mappings from Sparse Annotations »
Ashwin Kalyan · Stefan Lee · Anitha Kannan · Dhruv Batra