Unbiased Principles, Robust Rewards

Reward models are central to Reinforcement Learning from Human Feedback (RLHF), especially for open-ended tasks where evaluation is inherently multi-dimensional. Recent Generative Reward Models (GRMs) improve interpretability by producing natural-language rationales and task-specific evaluation principles. However, most existing GRMs generate principles after reading the actor's response, i.e., $Q+R \rightarrow P$. We show that this coupling induces Principle Drift: when the actor performs reward hacking (e.g., verbosity, self-aggrandizement, or hallucinated self-justifications), the reward model may shift its criteria to rationalize the response, yielding inflated scores that in turn reinforce hacking during RL. We propose IP-GRM (Independent Principle GRM), a two-stage framework that first generates principles solely from the question ($Q \rightarrow P$) and then evaluates the response conditioned on $(Q, R, P)$. This decoupling keeps criteria invariant to response content, producing more objective and stable reward signals. For efficient training, we further introduce a Principle Cache strategy that reuses principles within a group, improving GRPO throughput by 23.66\% while maintaining strict intra-group consistency. In GRPO training on creative writing, IP-GRM suppresses reward hacking and improves WritingBench and CreativeWriting-v3 by up to +4.6 and +7.1 points based on Qwen3-8B, achieving state-of-the-art performance among open-source models. The model and dataset are open-sourced at https://github.com/ShadeCloak/IP-GRM.

Modern AI assistants are often improved by having another AI act as a "judge" that grades their answers and rewards the good ones. We found a subtle but serious flaw in this setup: when the judge reads an answer first and then writes down what makes a "good" answer, it quietly bends its own standards to flatter whatever it just saw. The AI being trained quickly notices this loophole and learns to produce flashy or evasive answers that game the judge — instead of genuinely getting better. This is a well-known failure mode called reward hacking. We call the underlying cause principle drift, and we fix it with a simple change of order: ask the judge to write down its grading criteria from the question alone, before seeing any answer. Once the criteria are locked in, the judge then scores each answer against them — much like a teacher who writes the grading rubric before reading the essays. This small reordering keeps evaluation honest. AI models trained with our method are more stable during training, much harder to fool, and produce noticeably better answers on creative writing, safety, and reasoning tasks — a step toward AI systems that improve in directions we actually want.