Recontextualization Mitigates Specification Gaming Without Modifying the Specification

Developers often struggle to specify correct training labels and rewards. Perhaps they don't need to. We propose recontextualization, which reduces how often language models "game" training signals, performing misbehaviors those signals mistakenly reinforce. We show recontextualization prevents models from learning to 1) overfit evaluation criteria at the expense of chat response quality; 2) special-case code to pass incorrect tests; 3) overwrite evaluation functions rather than write correct code; and 4) become sycophantic. Our method works by generating completions from prompts discouraging misbehavior and then recontextualizing them as though they were in response to prompts permitting misbehavior. Recontextualization trains language models to resist misbehavior even when instructions permit it. This mitigates the reinforcement of misbehavior from misspecified training signals, reducing specification gaming without improving the supervision signal.

AI systems are often trained by rewarding answers that score well under a rule or evaluation process. A common problem is that they may learn to exploit loopholes in that rule instead of doing what people actually want. For example, they might give overly flattering answers, follow the letter of an instruction while missing its spirit, or find other shortcuts that receive high scores. We study a simple way to reduce this problem without changing the scoring rule. After the system generates answers, we change the instructions to appear more permissive than they originally were, making any loophole exploitation look like a response to the request rather than spontaneous behavior. Across several settings, this reduces unwanted shortcut-seeking while preserving the model’s ability to give useful answers. These results suggest that some training failures can be addressed not only by designing better scoring rules, but also by carefully choosing the situations in which models learn from those rules.