Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models

Post-hoc explanation methods are routinely used to interpret scientific machine learning models, with the deliverable understood to be insight into the phenomenon the model has been trained on. The transition may be taken to be secured once the model is reliable enough and the explanation faithful enough. We argue it is not. The two standards yield verdicts of different epistemic kinds, and their composition does not deliver a justified claim about the phenomenon's structure. The chain can support candidate hypotheses under external corroboration, but it cannot, on its own, support claims about how the phenomenon is in fact structured.