Timezone: »

 
Poster
Counterfactual Visual Explanations
Yash Goyal · Ziyan Wu · Jan Ernst · Dhruv Batra · Devi Parikh · Stefan Lee

Wed Jun 12 06:30 PM -- 09:00 PM (PDT) @ Pacific Ballroom #149
in Posters Wed »
In this work, we develop a technique to produce counterfactual visual explanations. Given a `query' image $I$ for which a vision system predicts class $c$, a counterfactual visual explanation identifies how $I$ could change such that the system would output a different specified class $c'$. To do this, we select a `distractor' image $I'$ that the system predicts as class $c'$ and identify spatial regions in $I$ and $I'$ such that replacing the identified region in $I$ with the identified region in $I'$ would push the system towards classifying $I$ as $c'$. We apply our approach to multiple image classification datasets generating qualitative results showcasing the interpretability and discriminativeness of our counterfactual explanations. To explore the effectiveness of our explanations in teaching humans, we present machine teaching experiments for the task of fine-grained bird classification. We find that users trained to distinguish bird species fare better when given access to counterfactual explanations in addition to training examples.
Keywords: Interpretability

Author Information

Yash Goyal (Georgia Tech)
Ziyan Wu (Siemens Corporate Technology)
Jan Ernst (Siemens Corporation)
Dhruv Batra (Georgia Institute of Technology / Facebook AI Research)
Devi Parikh (Georgia Tech & Facebook AI Research)
Stefan Lee (Georgia Institute of Technology)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors