Discounted Beta–Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards

Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective post-training paradigm for improving the reasoning capabilities of large language models. However, existing group-based RLVR methods often suffer from severe sample inefficiency. This inefficiency stems from reliance on point estimation of rewards from a small number of rollouts, leading to high estimation variance, variance collapse, and ineffective utilization of generated responses. In this work, we reformulate RLVR from a statistical estimation perspective by modeling rewards as samples drawn from a policy-induced distribution and casting advantage computation as the problem of estimating the reward distribution from finite data. Building on this view, we propose Discounted Beta–Bernoulli (DBB) reward estimation, which leverages historical reward statistics for the non-stationary distribution. Although biased, the resulting estimator exhibits reduced and stable variance and theoretically avoids variance collapse. Under mild non-stationarity, it also achieves a lower mean squared error than standard point estimation, as we characterize analytically and verify empirically. Across six in-distribution and three out-of-distribution reasoning benchmarks, GRPO with DBB consistently outperforms naive GRPO and strong recent baselines, including the replay-based RePO and the variance-collapse-aware GRESO and DAPO. Relative to GRPO, it achieves average Acc@8 improvements of 3.43/2.32 points in-distribution and 10.05/8.34 points out-of-distribution on the 1.7B and 8B models, respectively, without additional computational cost or memory usage.

Large language models are becoming better at solving complex reasoning problems, but training them to do so efficiently remains a challenge. A common approach, called reinforcement learning with verifiable rewards, improves models by rewarding correct answers. However, current training methods are wasteful because they rely on only a few generated examples at a time, making the learning signal noisy and unstable. In this work, we rethink the problem from a statistical perspective. Instead of treating each reward independently, we view rewards as samples from an evolving distribution that reflects the model’s behavior over time. Based on this idea, we introduce a new reward estimation method called Discounted Beta–Bernoulli (DBB), which reuses historical reward information while still adapting to changes during training. Although this introduces a small bias, it greatly stabilizes learning and reduces estimation errors. We provide both theoretical analysis and experimental evidence showing that our method avoids common training failures and makes better use of generated responses without requiring extra computation or memory. Across multiple reasoning benchmarks, DBB consistently improves performance over existing reinforcement learning approaches, including recent state-of-the-art methods. The gains are especially strong on challenging out-of-distribution tasks, showing that our approach helps language models generalize reasoning skills more effectively.