Process Reward Agents for Steering Knowledge-Intensive Reasoning

Reasoning in knowledge-intensive domains remains challenging as intermediate steps are often not locally verifiable: unlike math or code, evaluating step correctness may require synthesizing clues across large external knowledge sources. As a result, subtle errors can propagate through reasoning traces, potentially never to be detected. Prior work has proposed process reward models (PRMs), including retrieval-augmented variants, but these methods operate post hoc, scoring completed trajectories, which prevents their integration into dynamic inference procedures. Here, we introduce Process Reward Agents (PRA), an inference-time method for providing domain-grounded, online, step-wise rewards to a frozen policy. In contrast to prior retrieval-augmented PRMs, PRA enables search-based decoding to rank and prune candidate trajectories at every generation step. Experiments on multiple medical reasoning benchmarks demonstrate that PRA consistently outperforms strong baselines, achieving 81.9% accuracy on MedQA with Qwen3-4B, a new state of the art at the 4B scale. Importantly, PRA generalizes to unseen frozen policy models ranging from 0.5B to 8B parameters, improving their accuracy by up to 25.7% without any policy model updates. More broadly, PRA suggests a paradigm in which frozen reasoners are decoupled from domain-specific reward modules, allowing the deployment of new backbones in complex domains without retraining. All code and data are publicly available at https://process-reward-agents.github.io/.

AI models increasingly solve hard problems by reasoning one step at a time, including in medicine. But unlike math or computer code, where each step can be mechanically checked, medical reasoning is hard to verify: confirming whether a single step is correct can require piecing together evidence scattered across vast and constantly changing medical literature, so a subtle early mistake can quietly snowball through the whole chain of reasoning. Existing tools only grade a model's reasoning after it has finished, which is too late to catch the error. We built a separate AI that checks the work as it happens, looking over the reasoning model's shoulder one step at a time: at each step, it decides whether to look something up in trusted medical sources, judges whether the step holds up, and steers the model toward sounder lines of thought while discarding weak ones. Crucially, this checker needs no changes to the AI itself, so it can be attached to many different models without retraining. It improved accuracy across several medical exams (raising one small model's score by 25 points) and set a new record for compact models. Separating the fact-checker from the reasoner could make AI more trustworthy in high-stakes fields like healthcare.