Monitorability as a Free Gift: How RLVR Spontaneously Aligns Reasoning

As Large Reasoning Models (LRMs) are increasingly deployed, auditing their chain-of-thought (CoT) traces for safety becomes critical. Recent work has reported that monitorability—the degree to which CoT faithfully and informatively reflects internal computation—can appear as a "free gift" during the early stages of Reinforcement Learning with Verifiable Rewards (RLVR). We make this observation concrete through a systematic evaluation across model families and training domains. Our results show that this effect is not universal: monitorability improvements are strongly data-dependent. In particular, we demonstrate the critical role of data diversity and instruction-following data during RLVR training. We further show that monitorability is orthogonal to capability—improvements in reasoning performance do not imply increased transparency. Through mechanistic analysis, we attribute monitorability gains primarily to response distribution sharpening (entropy reduction) and increased attention to the prompt, rather than stronger causal reliance on reasoning traces. We also reveal how monitorability dynamics vary with controlled training and evaluation difficulty. Together, these findings provide a holistic view of how monitorability emerges under RLVR, clarifying when gains are likely to occur and when they are not.

Large Reasoning Models often show their “thinking” before giving a final answer. These written reasoning steps could help people check whether the AI is acting safely, honestly, or making a hidden mistake. However, this only works if the written reasoning truly reflects how the AI reaches its answer. This paper studies when AI reasoning becomes easier to check during reinforcement learning training. We find that this improvement does not always happen automatically. It depends strongly on what kind of data the AI is trained on. In particular, training with instruction-following data often makes the reasoning easier to inspect. We also find that better task performance does not necessarily mean the reasoning is more transparent. An AI can become better at solving problems while becoming no easier to audit. Overall, our findings help clarify how to train future AI systems whose reasoning is easier for humans and automated tools to check.