Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning

Recent advances in reinforcement learning (RL) have significantly improved the complex reasoning capabilities of large language models (LLMs). Despite these successes, existing methods mainly focus on single-domain RL (e.g., mathematics) with verifiable rewards (RLVR), and their reliance on purely online RL frameworks restricts the exploration space, thereby limiting reasoning performance. In this paper, we address these limitations by leveraging rubrics to provide both fine-grained reward signals and offline guidance. We propose $\textbf{RGR-GRPO}$ (Reward and Guidance through Rubrics), a rubric-driven RL framework for multi-domain reasoning. RGR-GRPO enables LLMs to receive dense and informative rewards while exploring a larger solution space during GRPO training. Extensive experiments across 14 benchmarks spanning multiple domains demonstrate that RGR-GRPO consistently outperforms RL methods that rely solely on alternative reward schemes or offline guidance. Compared with verifiable online RL baseline, RGR-GRPO achieves average improvements of +7.0%, +5.4%, +8.4%, and +6.6% on mathematics, physics, chemistry, and general reasoning tasks, respectively. Notably, RGR-GRPO maintains stable entropy fluctuations during off-policy training and achieves superior pass@k performance, reflecting sustained exploration and effective breakthrough beyond existing performance bottlenecks.

While large language models have become better at solving complex problems through trial-and-error learning, current methods often struggle when tasks move beyond simple math. Most existing systems only reward a model if the final answer is perfectly correct, which limits the model’s ability to explore different ways of thinking and solve problems in diverse fields like chemistry or physics. To bridge this gap, we developed RGR-GRPO, a new training framework that uses "rubrics"—detailed sets of scoring criteria—to guide the model. Instead of a simple "right or wrong" grade, our system provides fine-grained feedback throughout the reasoning process, acting like a teacher who gives partial credit and helpful hints. Our experiments across 14 different subject areas show that this approach significantly boosts performance, with improvements of up to 8.4% in complex science tasks. This research makes AI reasoning more robust and versatile, moving us closer to AI assistants that can tackle a wide range of sophisticated real-world challenges across multiple academic and professional domains.