Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents

In long-horizon tasks, recent agents based on Large Language Models (LLMs) face a significant challenge that sparse, outcome-based rewards make it difficult to assign credit to intermediate steps. Previous methods mainly focus on creating dense reward signals to guide learning, either through traditional reinforcement learning techniques like inverse reinforcement learning or by using Process Reward Models for step-by-step feedback. In this paper, we identify a fundamental problem in the learning dynamics of LLMs: the magnitude of policy gradients is inherently coupled with the entropy, which leads to inefficient small updates for confident correct actions and potentially destabilizes large updates for uncertain ones. To resolve this, we propose Entropy-Modulated Policy Gradients (EMPG), a framework that recalibrates the learning signal based on step-wise uncertainty and the final task outcome. EMPG amplifies updates for confident correct actions, penalizes confident errors, and attenuates updates from uncertain steps to stabilize exploration. We further introduce a bonus term for future clarity that encourages agents to find more predictable solution paths. Through comprehensive experiments on three challenging agent tasks, WebShop, ALFWorld, and Deep Search, we demonstrate that EMPG achieves substantial performance gains and significantly outperforms strong policy gradient baselines.

Training AI agents to complete long tasks is difficult because they often receive feedback only at the end. For example, an agent may browse a website, search for information, or interact with an environment through many steps, but it may only learn whether the final result was successful. This makes it hard to know which earlier actions should be encouraged or corrected. Our work studies how to make this end-of-task feedback more useful. We observe that an agent’s confidence at each step can provide a helpful clue. If the agent was confident and the final result was good, that step may deserve stronger encouragement. If the agent was confident but the final result was bad, it may need stronger correction. If the agent was uncertain, the training process should be more cautious. We turn this idea into a simple add-on for existing training methods. Across several benchmarks involving web interaction, household tasks, and search, our approach helps AI agents learn more stable and effective behaviors from limited feedback.