Exploration Hacking: Can LLMs Learn to Resist RL Training?

Reinforcement learning (RL) has become essential to the post-training of large language models (LLMs) for reasoning, agentic capabilities and alignment. Successful RL relies on sufficient exploration of diverse actions by the model during training, which creates a potential failure mode: a model could strategically alter its exploration during training to influence the subsequent training outcome. In this paper we study this behavior, called exploration hacking. First, we create model organisms of selective RL resistance by fine-tuning LLMs to follow specific underperformance strategies; these models can successfully resist our RL-based capability elicitation in agentic biosecurity and AI R&D environments while maintaining performance on related tasks. We then use our model organisms to evaluate detection and mitigation strategies, including monitoring, weight noising, and SFT-based elicitation. Finally, we show that current frontier models can exhibit explicit reasoning about suppressing their exploration when provided with sufficient information about their training context, with higher rates when this information is acquired indirectly through the environment. Together, our results suggest exploration hacking is a possible failure mode of RL on sufficiently capable LLMs.

AI developers increasingly train AI models using reinforcement learning, a method in which a model learns by trying many approaches and being updated towards the ones that work best. The method depends on the model genuinely exploring and trying to do well. This creates a possible weakness. A model could deliberately hold back during training, limiting its exploration so that the training fails to strengthen a particular ability. This behavior is called exploration hacking, and we present the first empirical study of it. We train AI models to underperform in specific, targeted ways, then test whether further reinforcement learning can recover their hidden abilities. Often it cannot. The models resist, especially when they follow a simple and predictable strategy, while keeping unrelated skills intact. This matters because the same kind of training is used to check whether an AI model has dangerous abilities, such as knowledge that could aid the creation of a weapon, before it is deployed. A model that can hide an ability could appear safe while concealing what it can really do. Encouragingly, the concealment is not hard to catch. Inspecting the model's internal reasoning reliably reveals it, and a different training method quickly reverses it. Today's most capable models can reason about this behavior when given enough context, but do not pursue it on their own. Together this suggests a real but currently manageable risk, and an opportunity to prepare countermeasures early.