DisPPO: Quantile-Based Distributional Reinforcement Learning for Large Language Models

Reinforcement Learning (RL) has become a cornerstone for enhancing the reasoning capabilities of Large Language Models (LLMs). However, standard actor-critic methods, such as PPO, rely on scalar value functions that estimate only the expectation of cumulative returns. This reduction inherently discards higher-order statistical information (e.g., variance and multimodality), leading to inaccurate value estimation and suboptimal credit assignment in complex tasks. While Distributional RL offers a solution by modeling the full return distribution, its application to LLMs remains challenging due to the computational intractability of value-based operations over large vocabularies and the instability and memory burden of off-policy replay mechanisms. In this paper, we propose DisPPO, a novel on-policy framework that seamlessly integrates non-parametric quantile regression into PPO. Theoretically, we prove that our distributional update operator---composed of the $\lambda$-return Bellman operator and quantile projection---is a contraction mapping in the Wasserstein metric, guaranteeing convergence to a unique fixed point. Empirically, we evaluate DisPPO using Llama and Qwen models across diverse benchmarks, including mathematical reasoning and Text-to-SQL generation. DisPPO consistently outperforms standard PPO and recent group-based baselines in both Pass@1 and Pass@$k$ metrics, demonstrating that distributional critics provide a richer, more robust learning signal for large-scale reasoning models.

Large language models are often trained to improve their reasoning by trying many answers and learning from which ones are correct. A common way to guide this training is to give each step a single score that estimates how useful it is. For hard problems, however, a single score can hide important uncertainty: the same partial answer may lead to several very different outcomes. We propose DisPPO, a training method that lets the model consider a range of possible future results instead of only one average estimate. This gives the model clearer feedback about which parts of its reasoning are likely to help. We test this idea on math problems, database query generation, multi-step tool use, and preference-based language model training. Across these settings, DisPPO improves performance over strong existing training methods while adding only a small amount of training cost and no extra cost when the model is used.