IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models

Generative Reward Models (GRMs) have demonstrated strong performance in reward modeling, due to their interpretability and potential for refinement through reinforcement learning (RL). However, widely used pairwise GRMs create a computational bottleneck in reinforcement learning from human feedback (RLHF), when calibrating or aggregating preference signals over $n$ candidates, often incurring $\mathcal{O}(n^2)$ pairwise judgments. To address this issue, we propose **Intergroup Relative Preference Modeling (IRPM)**, an RL-based method that extends the Bradley-Terry preference-learning paradigm via intergroup comparisons to train *pointwise* GRMs from pairwise preference data. IRPM derives a pointwise reward for each response by contrasting groups of chosen vs. rejected samples, enabling pointwise scores comparable across candidate sets and $\mathcal{O}(n)$ reward evaluation for a variable number of candidates during RL training, while preserving interpretability and scalability. Experiments show that IRPM achieves state-of-the-art performance among pointwise GRMs on JudgeBench and RewardBench, and approaches the performance of leading pairwise GRMs. In addition, IRPM achieves substantial gains in post-training evaluations, demonstrating its effectiveness. Our models and code are available at https://github.com/XINGYUN-AI-LAB/IRPM.

AI systems are often trained with the help of reward models, which act like judges that score how good a response is. The most accurate judges today usually compare answers in pairs, choosing which of two is better. This works well, but it is costly: evaluating 100 candidate answers can require nearly 5,000 pairwise comparisons. We propose IRPM, a method that uses only this comparison data to train a judge that can later score each answer independently. During training, the system generates multiple evaluations for both a preferred and a non-preferred answer, then cross-references the two groups to calibrate a consistent scoring standard. In experiments, IRPM matches the accuracy of strong pairwise methods at a fraction of the cost, while also allowing the judge to improve its accuracy by reasoning longer before giving a score. By enabling accurate independent scoring from cheap comparison data, IRPM lowers a key computational barrier in building AI systems that reliably follow human preferences.