Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis

Recent advances in reinforcement learning for code generation have made robust environments essential to prevent reward hacking. As LLMs increasingly serve as evaluators in code-based RL, their ability to detect reward hacking remains understudied. In this paper, we propose a novel taxonomy of reward exploits spanning across 54 categories and introduce TRACE (Testing Reward Anomalies in Code Environments), a synthetically curated and human-verified benchmark containing 517 testing trajectories. Unlike prior work that evaluates reward hack detection in isolated classification scenarios, we contrast these evaluations with a more realistic, contrastive anomaly detection setup on TRACE. Our experiments reveal that models capture reward hacks more effectively in contrastive settings than in isolated classification settings, with GPT-5.2 with highest reasoning mode achieving the best detection rate at 63%, up from 45% in isolated settings on TRACE. Building on this insight, we demonstrate that state-of-the-art models struggle significantly more with semantically contextualized reward hacks compared to syntactically contextualized ones. We further conduct qualitative analyses of model behaviors, as well as ablation studies showing that the ratio of benign to hacked trajectories and analysis cluster sizes substantially impact detection performance. We release the benchmark and evaluation harness to enable the community to expand TRACE and evaluate their models.

Recognizing when a model is taking undesired shortcuts or gaming tasks in coding environments is hard because of the scale at which model training is run. Observability over model actions in such cases is crucial and hence, automated evaluators or LLM judges are frequently used to guardrail model actions. In this study, we study how well these judges perform for coding executions. We found that, in isolation, LLM judges perform poorly at detecting hack patterns in code execution traces. We further found that this performance can be improved by analyzing multiple traces as a cluster contrastively and that this performance is best when there is a good sample of benign and gamed traces. After looking at predictions qualitatively, we found that semantic hack patterns in a cluster are much harder for the model to recognize than simple syntax hack patterns and that the judge can be influenced by factors such as user negligence or agent's false affirmations present in the trace.