Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RL

Maximum entropy has become a mainstream off-policy reinforcement learning (RL) framework for balancing exploitation and exploration. However, two bottlenecks still limit further performance gains: (1) non-stationary Q-value estimation stemming from the joint injection of entropy and the concurrent updating of its temperature parameter; and (2) short-sighted local entropy tuning, which adjusts temperature solely based on current single-step entropy without accounting for cumulative entropy over time. In this paper, we broaden the maximum entropy framework by proposing a trajectory entropy-constrained reinforcement learning (TECRL) framework to address these limitations. We begin by introducing reward-entropy separation (RES) to decouple the value targets, ensuring they remain stable and unaffected by temperature fluctuations. Subsequently, the resulting entropy Q-function is leveraged to explicitly quantify expected cumulative entropy, allowing for the enforcement of a trajectory entropy constraint (TEC) to govern long-term stochasticity. We instantiate this framework as DSAC-E, a practical off-policy algorithm that builds upon the latest distributional soft actor-critic. Extensive evaluations across 10 challenging tasks in locomotion, robotic manipulation, and vision-based driving domains demonstrate that DSAC-E consistently outperforms baselines in both cumulative returns and training stability.

Reinforcement learning is a technique that allows AI agents and robots to learn through trial and error. A widely used approach encourages agents to stay somewhat random during learning so they can continue exploring new behaviors instead of getting stuck too early. While this idea has been highly successful, existing methods still face two important problems: they can make unstable predictions during training, and they usually control randomness only based on the current moment without considering long-term behavior. In this work, we introduce a new framework that helps AI agents manage exploration more strategically over an entire sequence of decisions rather than one step at a time. Our method separates reward learning from randomness estimation, leading to more stable training and more reliable decision making. At the same time, it allows the agent to allocate its “exploration budget” more intelligently across different situations, instead of enforcing the same level of randomness everywhere. We develop a practical algorithm based on this framework and evaluate it on a diverse set of challenging tasks, including humanoid locomotion, robotic manipulation, and vision-based autonomous driving. The results show that our method consistently improves both learning stability and final performance compared with existing reinforcement learning approaches. These findings provide a more effective way for future AI systems and robots to learn complex behaviors safely and efficiently.