A Positive Case for Faithfulness: Explanations Help Predict Model Behavior

LLM self-explanations are often presented as a promising tool for AI oversight, yet their faithfulness to the model's true reasoning process is poorly understood. Existing faithfulness metrics have critical limitations, typically relying on identifying unfaithfulness via adversarial prompting or detecting reasoning errors. These methods overlook the predictive value of explanations. We introduce Normalized Simulatability Gain (NSG), a general and scalable metric based on the idea that a faithful explanation should allow an observer to learn a model's decision-making criteria, and thus better predict its behavior on related inputs. We evaluate 18 frontier proprietary and open-weight models, e.g., Gemini 3, GPT-5.2, and Claude 4.5, on 7,000 counterfactuals from popular datasets covering health, business, and ethics. We find self-explanations substantially improve prediction of model behavior (11-37% NSG). Self-explanations also provide more predictive information than explanations generated by external models, even when those models are stronger. This implies an advantage from self-knowledge that external explanation methods cannot replicate. Our approach also reveals that, across models, 5-15% of self-explanations are egregiously misleading. Despite their imperfections, we show a positive case for self-explanations: they encode information that helps predict model behavior.

LLMs can generate explanations of their decision-making when prompted; however, it is unclear to what extent these explanations correspond to the actual factors that influenced the decision, or whether the model fabricates a plausible-sounding explanation post hoc. This property is known as the faithfulness of the explanation. Previous work has proposed metrics for measuring explanation faithfulness. These have relied on biasing the model's decision-making process in a specific way, then assessing whether the model reports the influence of this bias in its explanation. Unfortunately, this approach does not scale to frontier models, which are more resistant to such biases. In this paper, we introduce a new metric for faithfulness based on a simple idea: a faithful explanation should help an observer predict how the model will behave on similar inputs. We test this by giving a model a question, recording its answer and explanation, and then asking another model to predict how it will answer a closely related counterfactual question, both with and without access to the explanation. If the explanation is faithful, access to it should improve this prediction. Across 18 leading LLMs and 7,000 counterfactual examples from health, business, and ethics datasets, we find that self-explanations consistently provide useful information about model behaviour. However, they are not perfect, and in some cases are actively misleading. Overall, our results make a positive case for LLM self-explanations.