Probing RLVR Training Instability through the Lens of Objective-Level Hacking

Prolonged reinforcement learning with verifiable rewards (RLVR) has been shown to drive continuous improvements in the reasoning capabilities of large language models, but the training is often prone to instabilities, especially in Mixture-of-Experts (MoE) architectures. Training instability severely undermines model capability improvement, yet its underlying causes and mechanisms remain poorly understood. In this work, we introduce a principled framework for understanding RLVR instability through the lens of objective-level hacking. Unlike reward hacking, which arises from exploitable verifiers, objective-level hacking emerges from token-level credit misalignment and is manifested as system-level spurious signals in the optimization objective. Grounded in our framework, together with extensive experiments on a 30B MoE model, we trace the origin and formalize the mechanism behind a key pathological training dynamic in MoE models: the abnormal growth of the training-inference discrepancy, a phenomenon widely associated with instability but previously lacking a mechanistic explanation. These findings provide a concrete and causal account of the training dynamics underlying instabilities in MoE models, offering guidance for the design of stable RLVR algorithms.

Training large language models to solve math and coding problems often relies on giving them rewards for correct answers and penalties for wrong ones. In principle, this should make the model better at reasoning, but in practice we find that this process may break down. Instead of continuing to improve, the model may lose reasoning ability in later stages of training, and in severe cases may lose its general language abilities, wasting large amounts of training computation. This work investigates why this happens. Through intervention experiments, we show that the problem is not simply random training noise, but a gradual shift in what the model is actually optimized to do. Some common training techniques can introduce hidden preferences into the optimization objective: while they help achieve their intended goals, their side effects can accumulate and push training in the wrong direction. By identifying this mechanism, our work provides a clearer explanation of a failure mode in large language model training and offers guidance for designing more stable and reliable training algorithms.