GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization

As language models become increasingly capable, users expect them to provide not only accurate responses but also behaviors aligned with diverse human preferences across a variety of scenarios. To achieve this, Reinforcement learning (RL) pipelines have begun incorporating multiple rewards, each capturing a distinct preference, to guide models toward these desired behaviors. However, recent work has defaulted to apply Group Relative Policy Optimization (GRPO) under multi-reward setting without examining its suitability. In this paper, we demonstrate that directly applying GRPO to normalize distinct rollout reward combinations causes them to collapse into identical advantage values, reducing the resolution of the training signal and resulting in suboptimal convergence and, in some cases, early training failure. We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues by decoupling the normalization of individual rewards, more faithfully preserving their relative differences and enabling more accurate multi-reward optimization, along with substantially improved training stability. We compare GDPO with GRPO across three tasks: tool calling, math reasoning, and coding reasoning, evaluating both correctness metrics (accuracy, bug ratio) and constraint adherence metrics (format, length). Across all settings, GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.

Modern AI assistants are taught complex behaviors through reinforcement learning, where the model tries many responses and is told which were better. Increasingly, researchers want models to balance several goals at once, for example being correct, concise, and well-formatted, by handing the model multiple reward signals simultaneously. The community has adopted a popular training recipe called GRPO for this setting, even though GRPO was originally designed for a single reward. We show that this casual reuse silently breaks the learning signal. When GRPO blends several rewards and then ranks responses within a batch, very different responses, say one that is accurate but verbose and another that is brief but wrong, can collapse to identical scores. The model is then told "these are equally good," loses the ability to tell behaviors apart, and training stalls or fails outright. We introduce GDPO, which keeps each reward on its own scale during this ranking step, so differences between behaviors are preserved. Across tool calling, math reasoning, and coding tasks, GDPO consistently beats GRPO on both correctness and adherence to formatting rules, and trains far more stably, giving practitioners a more reliable foundation for multi-objective AI training.