Thinking Reward Models Have a Self-Blind Spot: Chain-of-Thought Reward Models Cannot Detect Their Own Manipulation

Generative reward models (GenRMs) score responses by writing a chain-of-thought (CoT) critique before producing a numeric score, and have been proposed as a more robust replacement for scalar Bradley-Terry reward models. We find that adding the explicit CoT step does not strengthen the reward signal against simple manipulation; it weakens it. We train three reward models on identical UltraFeedback critique data -- a scalar BT RM, an answer-only GenRM, and a thinking GenRM that verbalizes a multi-step rubric -- and probe each with six surface-feature perturbations of the same base-policy responses. Five perturbations decrease scores across all three RMs; one -- appending a single confident closing sentence -- exclusively raises the thinking GenRM's score by +0.20 sigma (95% CI [+0.10, +0.30]) while leaving the scalar BT RM unchanged. We then ask the thinking GenRM explicitly whether the response is being gamed: this self-check raises the score on manipulated responses (delta = +0.25) more than on genuine ones (delta = +0.21), so a manipulated response that scored 8.54 before scores 8.79 after self-checking, ranking above the genuine response. Yet the same manipulation is trivially detectable from the outside: a Qwen2.5-3B classifier fine-tuned on ~1,400 examples reaches AUROC 0.973 / TPR@10%FPR 0.946, while using the thinking GenRM's own score as a detection signal yields AUROC 0.420 (below chance) and TPR@10%FPR 0.005. Together these three measurements identify a self-blind spot: the very component that is supposed to make GenRMs harder to fool -- the chain-of-thought rationale -- is exactly where the manipulation hides. We argue this is consistent with the unfaithful-CoT literature: the GenRM's rubric is post-hoc rationalization, not the actual computation that determines the score. We recommend not deploying a GenRM as both the scorer and its own gaming-detector, and propose external meta-evaluators as a cheap and effective complement.