The Unlearnability Phenomenon in RLVR for Language Models

Reinforcement Learning with Verifiable Reward (RLVR) has proven effective in improving Large Language Model's (LLM) reasoning ability. However, the learning dynamics of RLVR remain underexplored. In this paper, we reveal a counterintuitive phenomenon: among hard examples that the model initially struggles with, a substantial subset remains unlearnable even when correct rollouts are present. To understand the phenomenon, we first demonstrate that existing optimization and sampling techniques fail to resolve unlearnability. With cross-example gradient analysis, we show that unlearnable examples have fundamental representation issue, characterized by low gradient similarity with the rest of the examples and ungeneralizable reasoning patterns. We further show that representation flaws are difficult to mitigate in RL, as data augmentation does not improve gradient similarity. Our study provides the first systematic characterization of unlearnable data in RLVR training and reveals fundamental limitations in current RL approaches for reasoning tasks.

AI systems learn to reason by training on problems where they are rewarded for correct answers. The assumption is simple. If a model occasionally gets the right answer, repeated rewards will make it reliable. We show this breaks down for a substantial fraction of hard problems. Even when the model occasionally answers them correctly during training, it never actually learns to solve them, a phenomenon we call unlearnability. Common explanations, such as too few correct attempts, do not account for it. Instead, we find these problems are fundamentally different from the rest. The model lacks a coherent internal grasp of them, and its occasional correct answers come from flawed reasoning that happens to land on the right number. Generating similar practice problems does not help. Our findings reveal a basic limit of how AI is currently trained to reason. Rewarding correctness alone is not enough. Progress on the hardest problems likely requires improvements at earlier training stages that shape what the model internally understands.