From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Training

Reinforcement learning (RL) has become a widely adopted technique for improving large language models (LLMs) on complex tasks. Despite this progress, existing RL methods still face challenges in training agents with longer-horizon interactions. One major bottleneck is distinguishing the contribution of different actions in long-horizon interaction, leading to high optimization variance. To address this, we introduce a novel policy gradient method, Hindsight Policy Optimization (HPO), that projects both the current policy distribution and the hindsight distribution into an intent space and extracts low-variance learning signals from the Wasserstein distance between them. We theoretically and empirically show that aggregating semantically similar states and actions in the intent space yields a bounded-variance estimator and improves policy performance stably. Our code is available online.

How can language agents learn from long tasks when they only find out at the end whether they succeeded? In many real tasks, an agent must reason, search, and use tools over many steps, but the final reward gives little clue about which earlier decisions helped. This makes training unstable and inefficient, especially when successful attempts are rare. Considering that many different sentences in language actions can express the same underlying intent, we use meaning as a bridge for learning. For example, “Who is the president of the United States?” and “I want to know who the U.S. president is” use different words but ask for essentially the same thing. We propose Hindsight Policy Optimization, which maps actions into a semantic intent space and compares the model’s behavior trajectories with the patterns found in successful trajectories, helping identify which actions are truly valuable. This gives the model more useful step-by-step feedback, even when the original reward is only given at the end. By sharing learning signals across actions with similar meanings, HPO can reduce noisy training signals and help agents improve more reliably.