CausalXRL: Explainable Reinforcement Learning through Causal Graph Reasoning

Reinforcement learning is a powerful paradigm for training autonomous agents and has achieved impressive performance in complex environments. However, this success often comes at the cost of interpretability, diminishing trust and complicating efforts to debug and improve agent behavior. To address these challenges, we introduce CausalXRL, a novel framework for explainable reinforcement learning (XRL). A key feature of CausalXRL is its use of causal graph reasoning, which provides transparent, structured, multi-level explanations of agent decision-making. We validate CausalXRL through comprehensive case studies and a two-part evaluation: (1) a quantitative analysis of explanation fidelity and causal-structure learning efficiency in benchmark RL environments, and (2) a qualitative expert study assessing explainability in the real-time strategy (RTS) benchmark MicroRTS. The quantitative results show that CausalXRL can provide faithful explanations while efficiently learning causal structures, and the qualitative expert study suggests that participants found CausalXRL useful for inspecting high-level RTS strategies.

AI agents trained through trial and error can solve complex tasks, but their decisions are often difficult for people to understand. This is especially true in strategy games and other environments where many units, actions, and events interact at the same time. We developed CausalXRL, a method that converts an agent’s behavior into human-understandable cause-and-effect graphs. These graphs reveal both the agent’s overall strategy and the reasons behind specific decisions in individual situations. Unlike many existing explanation methods, CausalXRL helps people reason about how changes in one part of the environment may influence future agent behavior. This can help people inspect, debug, and build trust in AI systems. In our experiments, CausalXRL produced useful explanations across several reinforcement learning tasks, including a real-time strategy game evaluated by experienced human players.