One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models

Reward Models (RMs) are crucial for online alignment of language models (LMs) with human preferences. However, RM-based preference-tuning is vulnerable to \textit{reward hacking}, whereby LM policies learn undesirable behaviors from flawed RMs. By systematically measuring biases in five high-quality RMs, including the state-of-the-art, we find that issues persist despite prior work with respect to length, sycophancy, and overconfidence. We also discover new issues related to bias toward model-specific “styles” and answer-order. We categorize RM failures as tractable or resistant to linear intervention and propose a simple post-hoc intervention to mitigate low-complexity biases that arise from spurious correlations. Our proposed \textbf{mechanistic reward shaping} reduces targeted biases without degrading reward quality and while using minimal labeled data. The method is extensible to new biases, model-internal, and generalizes out-of-distribution.

Modern AI assistants like ChatGPT are trained to be helpful by using a second AI as a "judge" that scores their answers during training. These judges are supposed to abstract human values and other things we cannot naturally check or measure in math. If this judge has hidden biases, the assistant learns those biases too. We wanted to know how reliable today's judges actually are, and whether their flaws can be cheaply fixed without retraining. We tested five widely used judge models across six kinds of bias, including a preference for longer answers, agreeing with whatever the user said, sounding overconfident, and two we newly identified, favoring whichever answer appears first in a list and rewarding writing styles from particular AI model families. We found that some biases live as a single hidden pattern inside the judge, which we can mathematically remove. We call this mechanistic reward shaping. Other biases are tangled with the things the judge legitimately cares about and resist this simple fix. Our cheap correction reduces three kinds of bias without hurting the judge on standard tests, and works on new prompts it wasn't designed for. This gives practitioners a way to clean up known issues in existing judges, while also flagging which biases the field still has not solved.