Learning Interpretable Options by Identifying Reward Diffusion Bottlenecks in Reinforcement Learning

Bottleneck states, which connect distinct regions of the state space, provide a principled and interpretable basis for constructing temporal abstractions in Hierarchical Reinforcement Learning (HRL). However, existing bottleneck identification methods primarily rely on topological analysis of the state-transition graph, limiting their scalability to high-dimensional or continuous domains. To address this challenge, we introduce Value Power Strength (VPS), a value function-based metric inspired by the analogy between the Bellman equation and Kirchhoff’s current law, to quantify bottleneck property via the diffusion of reward in Markov Decision Processes (MDPs). VPS is estimated efficiently using value functions learned from random reward signals and captures reward diffusion bottlenecks in both discrete and continuous state spaces. Leveraging VPS, we design options that guide agents toward or away from bottleneck regions. Experimental results on classic tabular domains, continuous-control PointMaze, and Atari 2600 games demonstrate that the VPS-based framework discovers semantically meaningful subgoals and substantially improves exploration efficiency.

This paper studies how reinforcement learning agents can solve difficult tasks more efficiently by finding important “gateway” situations. In many environments, an agent must pass through certain key places or stages before it can make progress, such as a doorway between rooms, a key needed to open a door, or a narrow passage in a maze. Our method is inspired by electrical circuits. In a circuit, some points carry more flow or dissipate more energy because they connect important parts of the network. We use a similar idea to identify bottlenecks in learning environments: places where future rewards are strongly affected by passing through a particular state. Instead of building a full map of all possible states, our method detects these bottlenecks from the agent’s learned predictions. We then use the detected bottlenecks to create reusable behaviors that guide the agent toward or away from important regions. Experiments in gridworlds, a taxi task, a continuous maze, and Atari games show that our method finds meaningful intermediate goals and helps agents explore more effectively. This can make learning faster and the learned behaviors easier for humans to interpret.