Interpretability enriches what can be gleaned from a good predictive model. Techniques that learn-to-explain have arisen because they require only a single evaluation of a model to provide an interpretation. I will discuss a flaw with several methods that learn-to-explain: the optimal explainer makes the prediction rather than highlighting the inputs that are useful for prediction, and I will discuss how to correct this flaw. Along the way, I will develop evaluations grounded in the data and convey why interpretability techniques need to be quantitatively evaluated before their use.

References:

Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations: https://arxiv.org/pdf/2103.01890.pdf FastSHAP: Real-Time Shapley Value Estimation: https://arxiv.org/pdf/2107.07436.pdf Don't be fooled: label leakage in explanation methods and the importance of their quantitative evaluation: https://arxiv.org/pdf/2302.12893.pdf New-Onset Diabetes Assessment Using Artificial Intelligence-Enhanced Electrocardiography: https://arxiv.org/pdf/2205.02900.pdf