Rethinking the Trust Region in LLM Reinforcement Learning

Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm. Despite its ubiquity, we argue that the core ratio clipping mechanism in PPO is structurally ill-suited for the large vocabularies inherent to LLMs. PPO constrains policy updates based on the probability ratio of sampled tokens, which serves as a noisy single-sample Monte Carlo estimate of the true policy divergence. This creates a sub-optimal learning dynamic: updates to low-probability tokens are aggressively over-penalized, while potentially catastrophic shifts in high-probability tokens are under-constrained, leading to training inefficiency and instability. To address this, we propose Divergence Proximal Policy Optimization (DPPO), which substitutes heuristic clipping with a more principled constraint based on a direct estimate of policy divergence (e.g., Total Variation or KL). To avoid huge memory footprint, we introduce the efficient Binary and Top-K approximations to capture the essential divergence with negligible overhead. Extensive empirical evaluations demonstrate that DPPO achieves superior stability and efficiency compared to existing methods, offering a more robust foundation for RL-based LLM fine-tuning. Our code is available at \url{https://github.com/sail-sg/Stable-RL}.

Large language models are often improved after pretraining by letting them try answers and rewarding better ones, a process called reinforcement learning. The most widely used method, Proximal Policy Optimization (PPO), tries to keep each training step from changing the model too much by watching how the probability of the sampled next word changes. We find that this safeguard can be misleading for language models, because they choose from enormous vocabularies where some words are very likely and many are extremely rare. PPO can block useful changes to rare words even when the overall model behavior barely changes, while allowing risky changes to common words that can make training unstable. We propose Divergence Proximal Policy Optimization, or DPPO, which instead checks how much the model's whole prediction pattern has shifted. Because checking every word would be expensive, DPPO uses two lightweight approximations that capture the important changes with little extra cost. In experiments on math reasoning and on tasks where models learn to follow human preferences, DPPO trained models faster, avoided several training collapses, and reached better final performance than strong PPO-style baselines.