MRPO: Magnitude-Regularized Policy Optimization via L1 Constraints

Reinforcement learning (RL) for large language models (LLMs) relies on imperfect reward supervision, necessitating constraints on policy updates to prevent overfitting. Nevertheless, the widely adopted KL constraint over-penalizes actions with low reference probabilities and lacks the sparsity to discard marginal policy shifts. In contrast, the L1-norm offers a distinct mechanism that is more tolerant of low-probability actions yet strictly suppresses minor probability perturbations. Motivated by this, we propose Magnitude-Regularized Policy Optimization (MRPO), which enforces an L1-norm constraint on policy updates. We demonstrate that MRPO permits substantial probability boosts for low-probability actions and induces sparse updates, ensuring invariance to noise that preserves the top-ranking order. Furthermore, MRPO admits a TRPO-style monotonic improvement bound under standard regularity assumptions and achieves a tighter approach to optimality than KL-based methods in single-step scenarios. Empirically, MRPO delivers exceptional results across diverse scenarios, notably doubling the performance gains of GRPO in preference alignment, outperforming DAPO in mathematical reasoning, and surpassing DPO in offline settings using only binary rewards.

Many AI systems improve themselves by learning from feedback about what they did right or wrong. Today's methods also use safeguards to stop the model from changing too much and losing skills it already has. But the most widely used kind of safeguard can accidentally stop models from learning rare but useful behaviors. It can also make models easier to mislead with unclear feedback. We traced the math behind this problem and found a better safeguard. Instead of nudging many behaviors a little, our new safeguard leaves more behaviors untouched. It makes larger changes only to behaviors where the feedback is clear and strong. Surprisingly, changing only this safeguard is already enough to make existing learning methods work much better. To make the idea easy to use, we turned it into a clean algorithm called MRPO. It can help improve response quality and strengthen mathematical reasoning. In practice, MRPO is a simple drop-in replacement for a common part of existing methods, making it an easy upgrade.