QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning

GRPO-style reinforcement learning (RL)-based LLM fine-tuning algorithms have recently gained popularity. Relying on heuristic trust-region approximations, however, they can lead to brittle optimization behavior, as global importance-ratio clipping and group-wise normalization fail to regulate samples whose importance ratios fall outside the clipping range. We propose Query-Adaptive Trust-Region policy Optimization (QUATRO), which directly enforces trust-region constraints through a principled optimization. This yields a clear and interpretable objective that enables explicit control over policy updates and stable, entropy-controlled optimization, with a stabilizer terms arising intrinsically from the exact trust-region formulation. Empirically verified on diverse mathematical reasoning benchmarks, QUATRO shows stable training under increased policy staleness and aggressive learning rates, maintaining well-controlled entropy throughout training.

GRPO-style reinforcement learning (RL)-based LLM fine-tuning methods often rely on heuristic approximations to maintain training stability. However, these approximations can lead to unstable optimization and reduced output diversity. We propose QUATRO, a principled trust-region optimization method that directly constrains policy updates for each query. Unlike prior approaches based on heuristic clipping, QUATRO provides stable and interpretable optimization while maintaining well-controlled entropy during training. Experiments on mathematical reasoning benchmarks show that QUATRO remains stable even under aggressive learning rates and increased policy staleness, where existing methods often become brittle. These results suggest that principled trust-region optimization can make RL-based LLM fine-tuning more reliable and controllable for real-world applications.