Directly Optimizing Natural Language Explanations for Behavioral Faithfulness: Simulatability and Recoverability

Natural-language explanations are widely used to interpret machine learning models, yet many prioritize human plausibility over accurately reflecting or predicting model behavior. Prior approaches often rely on human-written rationales, producing post-hoc explanations that neither align with the model’s decision function nor generalize. We introduce $\textbf{OPeX}$, a natural-language explanation model that directly optimizes for behavioral faithfulness: the ability of an explanation to reflect and predict a model’s observable input–output behavior. $\textbf{OPeX}$ is trained using reinforcement learning with Group Relative Policy Optimization (GRPO), optimizing two complementary metrics: recoverability, which measures whether explanations recover model predictions on seen examples, and simulatability, which measures prediction of model behavior on unseen inputs. Across structured and text-based tasks, $\textbf{OPeX}$ achieves high simulatability (∼0.85) and recoverability (∼0.99), outperforming GPT-4o, LLaMA-3.3-70B, MaNtLE, Chain-of-Thought (CoT)-based models, and human-written explanations, despite using an 8B-parameter backbone. Human user studies show a 15% improvement in classification accuracy over competent baselines.

Natural-language explanations are widely used to help people understand machine learning models. However, many explanations prioritize sounding plausible over accurately reflecting a model’s behavior. As a result, they often fail to faithfully capture the model’s decision-making process and provide limited insight into how the model will behave in new situations. We introduce $\textbf{OPeX}$, an explanation generation model that produces explanations directly optimized to reflect the decision-making behavior of the machine learning model being explained. Unlike prior approaches that learn from human-written rationales, $\textbf{OPeX}$ explanations are directly optimized for $\textbf{behavioral faithfulness}$. $\textbf{OPeX}$ generates explanations that help users understand model behavior and predict how the model will behave in new scenarios. Across a diverse set of tasks, $\textbf{OPeX}$ produces more behaviorally faithful explanations than existing explanation methods and prominent thinking-based large language models (LLMs). User studies further show that these explanations help users better anticipate model predictions, suggesting that directly optimizing for behavioral faithfulness can lead to explanations that are both more accurate and more useful in practice.