The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probes

Training against white-box deception detectors has been proposed as a way to make AI systems honest. However, such training risks models learning to obfuscate their deception to evade the detector. Prior work has studied obfuscation only in artificial settings where models were directly rewarded for harmful output. We construct a realistic coding environment where reward hacking via hardcoding test cases naturally occurs, and show that obfuscation emerges in this setting. We introduce a taxonomy of possible outcomes when training against a deception detector. The model either remains honest, or becomes deceptive via two possible obfuscation strategies. (i) Obfuscated activations: the model outputs deceptive text while modifying its internal representations to no longer trigger the detector. (ii) Obfuscated policy: the model outputs deceptive text that evades the detector, typically by including a justification for the reward hack. Empirically, obfuscated activations arise from representation drift during RL, with or without a detector penalty. The detector penalty only incentivizes obfuscated policies; we theoretically show this is expected for policy gradient methods. Sufficiently high KL regularization and detector penalty can yield honest policies, establishing white-box deception detectors as viable training signals for tasks prone to reward hacking.

AI models trained with reinforcement learning often learn to "cheat" to get a high reward. For example, a coding AI might hardcode the answer to a test instead of actually solving the problem. We evaluate a proposed solution of using "lie detectors" that look at the model's internal activity for signs of deception, and penalize the model whenever the detector fires. We test this in a realistic coding setup where cheating emerges naturally, and find four possible outcomes: 1) the model becomes honest, 2) cheats openly and gets caught by the detector, 3) cheats while adapting its internals to no longer trigger the detector, or 4) cheats while writing elaborate justifications that fool the detector. We show that with the right balance of training penalties, lie detectors reliably push models toward honesty without hurting their problem-solving ability. This suggests lie detectors based on model internals are a viable way to train honest AI models.