Timezone: »

 
Don't trust your eyes: on the (un)reliability of feature visualizations
Robert Geirhos · Roland S. Zimmermann · Blair Bilodeau · Wieland Brendel · Been Kim

@
in 2nd ICML Workshop on New Frontiers in Adversarial Machine Learning »
Event URL: https://openreview.net/forum?id=s3GBSIakUy »

How do neural networks extract patterns from pixels? Feature visualizations attempt to answer this important question by visualizing highly activating patterns through optimization. Today, visualization methods form the foundation of our knowledge about the internal workings of neural networks, as a type of mechanistic interpretability. Here we ask: How reliable are feature visualizations? We start our investigation by developing network circuits that trick feature visualizations into showing arbitrary patterns that are completely disconnected from normal network behavior on natural input. We then provide evidence for a similar phenomenon occurring in standard, unmanipulated networks: feature visualizations are processed very differently from standard input, casting doubt on their ability to "explain" how neural networks process natural images. We underpin this empirical finding by theory proving that the set of functions that can be reliably understood by feature visualization is extremely small and does not include black-box neural networks.

Keywords: Deep Learning · Interpretability · explainability · Theory · Neural Networks · adversarial model manipulation · feature visualization · analysis · activation maximization

Author Information

Robert Geirhos (Google DeepMind)
Roland S. Zimmermann (University of Tübingen, MPI-IS)
Blair Bilodeau (University of Toronto)
Wieland Brendel (University of Tübingen)
Been Kim (Google Brain)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors