Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs

Reinforcement Learning (RL) with rubric-based rewards has recently shown remarkable progress in enhancing general reasoning capabilities of Large Language Models (LLMs), yet still suffers from ineffective exploration confined to current policy distribution. In fact, RL optimization can be viewed as steering the policy toward an ideal distribution that maximizes the rewards, while effective exploration should align efforts with desired target. Leveraging this insight, we propose HeRL, a **Hindsight experience guided Reinforcement Learning framework to bootstrap effective exploration by explicitly telling LLMs the desired behaviors specified in rewards. Concretely, HeRL treats failed attempts along with their unmet rubrics as hindsight experience, which serves as in-context guidance for the policy to explore desired responses beyond its current distribution. Additionally, we introduce a bonus reward to incentivize responses with greater potential for improvement under such guidance. HeRL facilitates effective learning from desired high-quality samples without repeated trial-and-error from scratch, yielding a more accurate estimation of the expected gradient theoretically. Extensive experiments across various benchmarks demonstrate that HeRL achieves superior performance gains over baselines, and can further benefit from experience guided self-improvement at test time.

Large language models can be improved with reinforcement learning, where they generate several answers to the same question and receive rewards that indicate which answers are better. However, a simple right-or-wrong reward gives little information about why an answer failed, so the model may struggle to discover better responses on its own.. Checklist-style rubrics provide more than scores: they describe what a good answer should include, avoid, or satisfy. Based on this observation, we propose HeRL, a reinforcement learning framework guided by hindsight experience. When an answer misses some rubric criteria, HeRL treats failed trajectories along with their unmet rubrics as hindsight experience. This helps the model learn what the reward standard expects more directly, instead of relying only on repeated trial and error. HeRL improves models more effectively than standard reinforcement learning baselines across different tasks.