Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization

Recent advancements in agentic test-time scaling allow models to gather environmental feedback before committing to final actions. A key limitation of existing methods is that they typically employ undifferentiated exploration strategies, lacking the ability to adaptively distinguish when exploration is truly required. In this paper, we propose an exploration-aware reinforcement learning framework that enables LLM agents to adaptively explore only when uncertainty is high. Our method introduces a fine-grained reward function via variational inference that explicitly evaluates exploratory actions by estimating their potential to improve future decision-making, together with an exploration-aware grouping mechanism that separates exploratory actions from task-completion actions during optimization. By targeting informational gaps, this design allows agents to explore selectively and transition to execution as soon as the task context is clear. Empirically, we demonstrate that our approach achieves consistent improvements across a range of challenging text-based and GUI-based agent benchmarks.

Current AI agents are largely goal-oriented: they focus on directly completing tasks, but when they encounter unfamiliar situations, they often fail to actively try possible solutions or gather new information. For example, in scientific research, a researcher may test different hypotheses, explore alternative experiments, or search for missing evidence before reaching a conclusion. Existing AI agents, however, tend to follow a fixed strategy instead of adaptively exploring uncertain situations. Our paper introduces a new training framework that teaches AI agents to explore selectively and learn from the information they discover. We design a reward system that evaluates whether an exploratory action improves future decision-making, allowing the agent to distinguish between situations that require exploration and those where direct action is sufficient. Our experiments show that this approach substantially improves performance on challenging web, mobile, and computer interaction tasks, especially in long and unfamiliar environments. The results suggest that effective AI agents should not only pursue goals, but also learn how to intelligently explore when they lack sufficient information.