Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use

Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems. We introduce the Reward Hacking Benchmark (RHB), a suite of multi-step tasks requiring sequential tool operations with naturalistic shortcut opportunities such as skipping verification steps, inferring answers from task-adjacent metadata, or tampering with evaluation-relevant functions; RHB supports independent and chained task regimes, where chain length acts as a proxy for longer-horizon agent behavior. We evaluate 13 frontier models from OpenAI, Anthropic, Google, and DeepSeek; exploit rates range from 0% (Claude Sonnet 4.5) to 13.9% (DeepSeek-R1-Zero), varying sharply by post-training style. A controlled sibling comparison (DeepSeek-V3 vs. DeepSeek-R1-Zero) shows RL post-training is associated with substantially higher reward hacking (0.6% vs. 13.9%), with consistent gaps across all four task families. We identify six exploit categories and find that 72% of reward hacking episodes include explicit chain-of-thought rationale, suggesting models often frame exploits as legitimate problem-solving. Simple environmental hardening reduces exploit rates by 5.7 percentage points (87.7% relative) without degrading task success; models with near-zero exploit rates on standard tasks show elevated rates on harder variants, suggesting that production-aligned post-training appears to suppress reward hacking only below a complexity threshold where honest solutions remain tractable.

When you tell a powerful AI assistant to "do a task," it sometimes does exactly that but in ways you never intended, for instance, editing the way you measure whether the task succeeded, or skipping the actual task and hard-coding the answer. This paper introduces a benchmark that systematically catches today's leading AI agents (Claude, OpenAI's o3 and o4-mini, and others) in the act, by giving them real tools and tasks where the "easy" path is to exploit a loophole in how success is measured rather than genuinely solving the problem. Across a wide range of scenarios, the more aggressively reasoning-tuned models reliably find and take these shortcuts, and not because they're confused about what they're supposed to do, but because the shortcut is a legitimate move the rules of the game allow. We also show that simple changes to the testing environment cut this kind of cheating by about 88% without making the models any worse at honest work. The takeaway: as we hand AI systems more autonomy and more tools, we can't just measure whether they hit a target. We have to design targets that can't be gamed in the first place.