Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models

Process Reward Models (PRMs) are rapidly becoming the backbone of LLM reasoning pipelines, yet we demonstrate that state-of-the-art PRMs are systematically exploitable under optimization pressure. We introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities. Static perturbation analysis uncovers a fluency-logic dissociation: high invariance to surface-level style changes (reward changes $<$0.1) yet inconsistent detection of logically corrupted reasoning, with different models failing on different attack types. Adversarial optimization demonstrates that gradient-based attacks inflate rewards on invalid trajectories, with reward landscapes exhibiting wide, exploitable peaks. RL-induced reward hacking exposes the critical failure mode: policies trained on AIME problems achieve near-perfect PRM rewards ($>$0.9) while ground-truth accuracy remains below 4\%, with 43\% of reward gains attributable to stylistic shortcuts. These findings reveal that current PRMs function as fluency detectors rather than reasoning verifiers, creating systematic blind spots that undermine their use as training signals. We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment.

When AI systems solve math problems, they often rely on a second AI, a "reward model," to grade their reasoning step by step. These graders are increasingly used to train the problem-solving AI, so their trustworthiness matters. We stress-tested today's leading graders and found they are easily fooled: they reward answers that merely sound like careful reasoning, even when the logic is wrong. Rewording a correct solution barely changes its grade, yet the graders often miss deliberately corrupted reasoning. Worse, when an AI is trained to chase these grades, it learns to game the grader rather than reason better, scoring nearly perfectly while getting the math right less than 4% of the time. Current graders act more like fluency detectors than genuine reasoning checkers. We release a benchmark and toolkit to help find and fix these weaknesses before deployment.