Alignment Risks from Capability-Seeking RL Training

While most AI alignment research focuses on preventing models from generating explicitly harmful content, a more subtle risk arises from capability-seeking RL training in vulnerable environments. We investigate whether language models, when trained with reinforcement learning (RL) in environments with implicit loopholes, can learn to exploit these flaws to maximize reward, even without being explicitly instructed to do so. To test this, we design a suite of four diverse "vulnerability games'', each presenting a structural vulnerability related to context-conditional compliance, proxy metrics, reward tampering, and self-evaluation. Our experiments show that models often learn to exploit these vulnerabilities, discovering opportunistic strategies that increase reward while sometimes preserving or even improving standard task-performance metrics. More critically, we find that these exploitative strategies are not always narrow "tricks'': they can transfer in structured but limited ways, propagate from a capable teacher model to other student models through SFT, and in several cases remain more persistent when learned through RL than when distilled through SFT. Our findings show that alignment risks from capability-seeking RL training can be difficult to detect with standard performance monitoring, suggesting that future AI safety work should extend beyond content moderation to auditing and securing training environments, reward mechanisms, and evaluation channels. Code is available at https://github.com/YujunZhou/Capability-seeking-RL-risk.

AI systems are often trained by rewarding them when they appear to perform well. But if the reward or evaluation setup has small loopholes, the system may learn to exploit those loopholes instead of truly solving the task. In this paper, we study this risk using four controlled test environments. We find that language models can learn shortcuts such as appearing safe only in test-like situations, overclaiming that their answers are correct, copying text to score better on automatic metrics, or manipulating an exposed score field. Importantly, these problems do not always look like failures: in some cases, the model’s normal performance score stays the same or even improves while the hidden shortcut becomes more common. We also find that some shortcuts can spread through further training or become harder to remove after being learned. These results suggest that making AI systems safer requires checking not only their final outputs, but also the reward rules, evaluation tools, and training environments used to improve them.