UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms

Reinforcement learning (RL) excels on tasks with verifiable rewards, but in open-ended tasks, the reliability of reward models remains a key challenge. Existing solutions either depend on costly proprietary LLM-as-a-Judge systems or opaque scalar reward models that lack interpretability. Recent works on generative reward models offer a promising alternative, but they remain constrained by static evaluation criteria, fragmented evaluation paradigms, and limited multilingual support. To address these challenges, we introduce MixReward, a large-scale multilingual dataset spanning six domains and 103 languages, containing both pairwise and listwise data, and propose UniRRM, a unified reasoning reward model supporting multiple languages and evaluation paradigms. UniRRM uses a staged reasoning chain to dynamically generate task-generic and instruction-specific criteria, enabling fine-grained, input-adaptive judgments while maintaining consistency across languages. Experiments demonstrate that UniRRM-8B and UniRRM-14B achieve performance close to the state-of-the-art for models of comparable size across multiple benchmarks, and are effective for unseen evaluation paradigms. In addition, ablation studies validate the reliability and effectiveness of UniRRM.

Recent advances in reinforcement learning have enabled AI systems to perform remarkably well on tasks with clear right or wrong answers, such as solving math problems or writing code. However, many real-world tasks—like evaluating helpfulness, creativity, or reasoning quality—do not have straightforward rewards. In these open-ended settings, AI systems rely on “reward models” to judge response quality, but existing approaches often depend on expensive proprietary systems or black-box scoring models that provide little transparency into how decisions are made. To address these limitations, we introduce MixReward and UniRRM, a unified multilingual reasoning reward framework designed to make AI evaluation more transparent, adaptable, and broadly accessible. MixReward is a large-scale multilingual dataset covering 103 languages and multiple evaluation settings, enabling reward models to learn from diverse human preferences across many domains. Building on this dataset, UniRRM generates explicit reasoning chains and dynamically adapts its evaluation criteria based on the input task and instructions, rather than relying on fixed scoring rules. Our approach allows the reward model to provide more fine-grained and interpretable judgments while maintaining consistency across languages and evaluation formats. Experimental results show that UniRRM achieves competitive performance with state-of-the-art reward models of similar size and generalizes effectively to previously unseen evaluation settings. These findings suggest that transparent and multilingual reasoning-based reward models can serve as a practical alternative to proprietary AI judges, helping make future AI systems more reliable, interpretable, and globally accessible.