Optimizing Agentic Reasoning with Retrieval via Synthetic Semantic Information Gain Reward

Agentic reasoning enables large reasoning models (LRMs) to dynamically acquire external knowledge, yet optimizing the retrieval process remains challenging due to the lack of dense, principled reward signals. In this paper, we introduce InfoReasoner, a unified framework that incentivizes effective information seeking via a synthetic semantic information gain reward. Theoretically, we redefine information gain as uncertainty reduction over the model's belief states, establishing key properties including non-negativity, telescoping additivity, and channel monotonicity. Practically, to enable scalable optimization without manual intermediate retrieval annotations, we instantiate this principle as a semantic information gain reward computed from the model's output distributions using semantic clustering via bidirectional textual entailment. This training reward provides dense credit for retrieval steps while remaining anchored to final-answer correctness, enabling efficient training via Group Relative Policy Optimization (GRPO). Experiments on seven question-answering benchmarks, MATH500, and WebDetective show consistent gains over strong retrieval-augmented baselines, supporting our dense semantic information gain as a practical training signal for agentic retrieval.

Modern AI systems often need to look up information while solving difficult questions, but it is hard to teach them which search actions are actually useful. If we only reward the final answer, the system receives little guidance about whether each intermediate search helped or wasted effort. We introduce InfoReasoner, a training framework that gives the model a more informative learning signal. The key idea is simple: a good search result should make the model less uncertain about the correct answer. To measure this, InfoReasoner samples possible answers from the model, groups answers with the same meaning, and rewards searches that increase the model’s confidence in the correct meaning. This lets the model learn better search and reasoning behavior without requiring humans to label every intermediate retrieval step. Across question answering, mathematical reasoning with tools, and long-horizon web search tasks, InfoReasoner improves accuracy over strong retrieval-based baselines. This work helps build AI agents that search more purposefully, use evidence more effectively, and give more reliable answers.