From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning

Reinforcement learning has become a cornerstone for enhancing the reasoning capabilities of Large Language Models, where group-based approaches such as GRPO have emerged as efficient paradigms that optimize policies by leveraging intra-group performance differences. However, these methods typically rely on absolute numerical rewards, introducing intrinsic limitations. In verifiable tasks, identical group evaluations often result in sparse supervision, while in open-ended scenarios, the score range instability of reward models undermines advantage estimation based on group means. To address these limitations, we propose Reinforcement Learning with Relative Rewards (RLRR), a framework that shifts reward shaping from absolute scoring to relative ranking. Complementing this framework, we introduce the Ranking Reward Model, a listwise preference model tailored for group-based optimization to directly generate relative rankings. By transforming raw evaluations into robust relative signals, RLRR effectively mitigates signal sparsity and reward instability. Experimental results demonstrate that RLRR yields consistent performance improvements over standard group-based baselines across reasoning benchmarks and open-ended generation tasks.

Large language models have become remarkably good at solving math problems and writing code, partly because of a training method called reinforcement learning. In this process, the model tries many different answers to the same question, and a scoring system rewards the better ones so the model learns to improve. However, a popular training technique called GRPO has a key weakness: it relies on exact numerical scores to compare answers within a group. When most answers in a group are either all correct or all wrong, the scores become identical, and the model receives no useful feedback to keep learning. This wastes a large portion of training effort as the model gets better and more groups become uniformly correct. We propose a method called RLRR that replaces exact numerical scores with relative rankings. Instead of asking "how good is this answer?", we ask "which answer is better than the others?" A small ranking model learns to order multiple answers from best to worst, providing a stable signal that does not depend on fragile score magnitudes. This ranking-based feedback continues to distinguish answers even when binary correctness scores cannot. Our experiments show that RLRR consistently improves reasoning performance across math and coding benchmarks while making better use of the training data. This approach suggests that teaching AI through relative comparisons, much like how humans learn by comparing options, can be more effective than relying on absolute scores.