Self-evolving LLM agents with in-distribution Optimization

Large Language Models (LLMs) have recently emerged as powerful controllers for interactive agents in complex environments, yet training them to perform reliable long-horizon decision making remains a fundamental challenge. A key difficulty lies in credit assignment: agents often receive delayed rewards only at the end of episodes. In this paper, we propose Q-Evolve, a self-evolving framework for LLM agents that unifies automatic process-reward labeling and policy learning within a principled in-distribution reinforcement learning paradigm. In each evolving iteration, our method learns an in-distribution critic from a hybrid off-policy dataset that combines expert demonstrations with agent-generated trajectories, stabilizing Bellman backups in sparse-reward settings via a weighted Implicit Q-Learning objective. The learned value function is then used to derive step-wise process rewards through advantage estimation, enabling dense and reliable supervision without environment backtracking or human annotation. Leveraging these signals, we perform behavior-proximal policy optimization that evolves the agent over the data used for process reward labeling, allowing iterative self-improvement without exacerbating distribution shift. We evaluate our method on AlfWorld, WebShop, and ScienceWorld, showing Q-Evolve outperforms strong baselines in sample efficiency, robustness, and overall task performance. Our results demonstrate that stable agent self-evolution is achievable through the co-evolution of process-level supervision and policy, both grounded within a shared in-distribution learning loop.

Many real-world AI agents need to make decisions over many steps. Giving each step a useful score can make training much easier and stable, but these scores can become unreliable when they are applied to situations the reward model has never seen. To this end, we introduce Q-Evolve, a framework that avoids this problem by computing step-level training signals and improving the agent on the same set of collected experiences. The agent learns from both expert examples and its own experience, estimates which steps helped or hurt the final outcome through the process reward models (a Q model), and then updates its behavior while staying close to the training data of the Q model. This shared-data design makes self-improvement in-distribution, thus more stable and reliable. Experiments show that Q-Evolve helps AI agents perform better on complex interactive tasks with better sample efficiency.