Automatically Finding Reward Model Biases

Reward models are central to large language model (LLM) post-training. However, past work has shown that they can reward spurious or undesirable attributes such as length, format, hallucinations, and sycophancy. In this work, we introduce and study the research problem of automatically finding reward model biases in natural language. We offer a simple approach of using an LLM to iteratively propose and refine candidate biases. Our method can recover known biases and surface novel ones: for example, we found that Skywork-V2-8B, a leading open-weight reward model, often mistakenly favors responses with redundant spacing and responses with hallucinated content. In addition, we show evidence that evolutionary iteration outperforms flat best-of-N search, and we validate the recall of our pipeline using synthetically injected biases. We hope our work contributes to further research on improving RMs through automated interpretability methods.

Many AI systems are trained using "reward models”, which are programs that score answers so that the AI can learn to give responses people prefer. But these reward models can make systematic mistakes: they may give higher scores to answers that look good on the surface but are misleading, poorly formatted, or otherwise worse for users. This paper studies how to automatically find these hidden mistakes before they affect the final AI system. We build a method where one AI proposes possible flaws in a reward model, tests whether the reward model actually favors those flaws, and then refines the most promising ideas over several rounds. Using this approach, we found that a strong open reward model sometimes preferred answers with problems such as unnecessary extra spaces, unsupported quotes, made-up details, or misleading wording. Our results suggest that automated audits can help developers catch reward model failures earlier and make AI training more reliable.