Redundancy Undermines the Trustworthiness of Self-Interpretable GNNs

This work presents a systematic investigation into the trustworthiness of explanations generated by self-interpretable graph neural networks (GNNs), revealing why models trained with different random seeds yield inconsistent explanations. We identify redundancy—resulting from weak conciseness constraints—as the root cause of both explanation inconsistency and its associated inaccuracy, ultimately hindering user trust and limiting GNN deployment in high-stakes applications. Our analysis demonstrates that redundancy is difficult to eliminate; however, a simple ensemble strategy can mitigate its detrimental effects. We validate our findings through extensive experiments across diverse datasets, model architectures, and self-interpretable GNN frameworks, providing a benchmark to guide future research on addressing redundancy and advancing GNN deployment in critical domains. Our code is available at \url{https://github.com/ICDM-UESTC/TrustworthyExplanation}.

Graph neural networks (GNNs) are powerful tools used in areas like healthcare, finance, and scientific discovery. In many of these high-stakes applications, it’s important not only that a model makes accurate predictions, but also that it can explain why it made those predictions. Some GNNs are designed to be self-explaining—that is, they provide their own reasons for each decision. However, our study finds that these explanations can vary significantly when the model is trained multiple times, even with the same data. This inconsistency can confuse users and reduce trust in the system. We find that the root cause of this issue is redundancy in the explanations—extra or unnecessary parts that don’t help with understanding the decision. While it’s hard to eliminate this redundancy completely, we show that using a simple ensemble (combining multiple models) can help improve explanation consistency and its associated accuracy. Our findings help pave the way for safer and more trustworthy GNN use in real-world, high-impact domains.