Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration

Search-integrated reasoning enables language agents to transcend static parametric knowledge by actively querying external sources. However, training these agents via reinforcement learning is hindered by the multi-scale credit assignment problem: existing methods typically rely on sparse, trajectory-level rewards that fail to distinguish between high-quality reasoning and fortuitous guesses, leading to redundant or misleading search behaviors. To address this, we propose Search-R2, a novel Actor–Refiner collaboration framework that enhances reasoning through targeted intervention, with both components jointly optimized during training. Our approach decomposes the generation process into an Actor, which produces initial reasoning trajectories, and a Meta-Refiner, which selectively diagnoses and repairs flawed steps via a ``cut-and-regenerate'' mechanism. To provide fine-grained supervision, we introduce a hybrid reward design that couples outcome correctness with a dense process reward quantifying the information density of retrieved evidence. Theoretically, we formalize the Actor–Refiner interaction as a smoothed mixture policy, proving that selective correction yields strict performance gains over strong baselines. Extensive experiments across various general and multi-hop QA datasets demonstrate that Search-R2 consistently outperforms strong RAG and RL-based baselines across model scales, achieving superior reasoning accuracy with minimal overhead.

AI agents can now look up information while answering questions, which helps them go beyond what they already know. However, when an agent searches for the wrong thing early on, it may follow that misleading information and make a wrong final answer. Current training methods often only check whether the final answer is correct, so they may miss where the agent’s reasoning first went off track. We introduce Search-R2, a training method that teaches an AI agent to notice and fix mistakes during search-based reasoning. Instead of throwing away a whole failed answer, Search-R2 keeps the useful earlier steps, cuts off the part where the mistake begins, and lets the agent try again from that point. It also rewards answers that are not only correct, but supported by useful retrieved information. Across several question-answering tasks, Search-R2 improves answer accuracy and produces more reliable search behavior, while adding little extra training cost and no extra cost when the agent is used. This work is a step toward AI agents that search more carefully, recover from misleading information, and give more trustworthy answers.