This State Looks Like That: Self-Interpretable Reinforcement Learning Agents using Prototype Soft Actor-Critic

Reinforcement learning (RL) has achieved remarkable success across complex decision-making tasks, especially with the advent of deep neural networks. However, the resulting models are often opaque, making their deployment in safety-critical domains challenging. Explainable AI aims to address this issue, but most specific efforts for deep RL remain limited either to post-hoc explanation methods or to imitation learning and distillation procedures. These latter approaches rely on pre-trained black-box agents and are typically restricted to environments with discrete action spaces, limiting their scalability and interpretability. In this paper, we introduce ProtoSAC, a novel deep RL architecture that integrates a prototype-based actor into the Soft Actor-Critic (SAC) algorithm, enabling intrinsic interpretability in continuous action spaces. Our method learns a set of prototypes that represent interpretable state clusters, each associated with a Gaussian action distribution. Actions are generated as a similarity-weighted mixture over these prototypes, providing more inspectable and decomposable decision-making without sacrificing performance compared to standard SAC. We evaluate ProtoSAC on continuous control environments and show that it matches the performance of the original SAC while offering enhanced interpretability.

AI systems that learn through trial and error — known as reinforcement learning agents — have become remarkably capable, from controlling robots to playing complex games. But there is a catch: we rarely understand why they make the decisions they do. This opacity is a serious problem in safety-critical settings, such as autonomous vehicles or medical systems, where trust and accountability matter. We developed ProtoSAC, an AI agent that reasons by comparing its current situation to a set of learned prototypes: representative examples of past experiences. Rather than producing a decision through an inscrutable chain of computations, the agent can say, in effect, "this situation looks like that familiar one, so I'll act accordingly." This makes its reasoning directly inspectable. Tested across eight continuous control tasks, from balancing a pendulum to landing a lunar module, ProtoSAC matches the performance of standard black-box agents while offering significantly more transparent and faithful explanations of its behavior.