Optimizing Explanations: Nuances Matter When Evaluation Metrics Become Loss Functions

Recent work has introduced a framework that allows users to directly optimize explanations for desired propertiesand their trade-offs. While powerful in principle, this method repurposes evaluation metrics as loss functions – anapproach whose implications are not yet well understood. In this paper, we study how different robustness metricsinfluence the outcome of explanation optimization, holding faithfulness constant. We do this in the transductivesetting, in which all points are available in advance. Contrary to our expectations, we observe that the choice ofrobustness metric can lead to highly divergent explanations, particularly in higher-dimensional settings. We tracethis behavior to the use of metrics that evaluate the explanation set as a whole, rather than imposing constraints onindividual points, and to how these “global” metrics interact with other optimization objectives. These interactionscan allow the optimizer to produce locally inconsistent, unintuitive, and even undesirable explanations, despitesatisfying the desired trade-offs. Our findings highlight the need for metrics whose mathematical structure moreclosely aligns with their intended use in optimization, and we advocate for future work that rigorously investigatesmetrics that incorporate a pointwise evaluation and their influence on the optimization landscape.