T$^2$PO: Uncertainty-Guided Exploration Control for Stable Multi-Turn Agentic Reinforcement Learning

Recent progress in multi-turn reinforcement learning (RL) has significantly improved reasoning LLMs' performances on complex interactive tasks. Despite advances in stabilization techniques such as fine-grained credit assignment and trajectory filtering, instability remains pervasive and often leads to training collapse. We argue that this instability stems from inefficient exploration in multi-turn settings, where policies continue to generate low-information actions that neither reduce uncertainty nor advance task progress. To address this issue, we propose Token- and Turn-level Policy Optimization (T$^2$PO), an uncertainty-aware framework that explicitly controls exploration at fine-grained levels. At the token level, T$^2$PO monitors uncertainty dynamics and triggers a thinking intervention once the marginal uncertainty change falls below a threshold. At the turn level, T$^2$PO identifies interactions with negligible exploration progress and dynamically resamples such turns to avoid wasted rollouts. We evaluate T$^2$PO in diverse environments, including WebShop, ALFWorld, and Search QA, demonstrating substantial gains in training stability and performance improvements with better exploration efficiency. Code is available at https://github.com/WillDreamer/T2PO.

Large language model agents are increasingly trained to solve tasks by reasoning, taking actions, and learning from feedback over many steps. However, this training can be unstable: agents may keep producing long, repetitive thoughts or unhelpful actions that add noise instead of helping them solve the task. We introduce T$^2$PO, a training method that watches the model’s own uncertainty while it is generating responses. When the model’s thinking stops changing in a meaningful way, T2$^2$O ends the thinking phase and moves it to action. When an entire interaction turn appears to repeat the same low-value pattern, T$^2$PO asks the agent to try that turn again. This helps the agent spend less time on redundant exploration and more time on useful decisions. We tested T$^2$PO on WebShop, AlfWorld, and search-based QA tasks. Across these settings, it improved success rates, reduced wasted tokens and turns, and made training more stable. Our results suggest that controlling when agents think and when they act can make AI agents more reliable and efficient.