Asymmetric Prompt Weighting for Reinforcement Learning with Verifiable Rewards

Reinforcement learning with verifiable rewards has driven recent advances in LLM post-training, in particular for reasoning. Policy optimization algorithms generate a number of responses for a given prompt and then effectively weight the corresponding gradients depending on the rewards. The most popular algorithms including GRPO, DAPO, and RLOO focus on ambiguous prompts, i.e., prompts with intermediate success probability, while downgrading gradients with very easy and very hard prompts. In this paper, we consider asymmetric prompt weightings that assign higher weights to prompts with low, or even zero, empirical success probability. We find that asymmetric weighting particularly benefits from-scratch RL (as in R1-Zero), where training traverses a wide accuracy range, and less so in post-SFT RL where the model already starts at high accuracy. We also provide theory that characterizes prompt weights which minimize the time needed to raise success probability from an initial level to a target accuracy under a fixed update budget. In low-success regimes, where informative responses are rare and response cost dominates, these optimal weights become asymmetric, upweighting low success probabilities and thereby accelerating effective-time convergence.

When training AI systems to solve math or coding problems, a common approach is to have the model generate many attempted solutions, check which ones are correct, and update the model based on these results. Current methods focus their learning effort on problems of medium difficulty (ones the model sometimes gets right and sometimes gets wrong) while largely ignoring problems the model almost never solves. This means the hardest problems receive almost no learning signal, which can limit the model's final performance. We propose a simple change: give harder problems more weight during training. Our asymmetric re-weighting methods ensure that even problems the model currently fails mostly or entirely still contribute a useful learning signal. In experiments where a model starts with near-zero ability and must learn to reason from scratch, our approach reaches about 80% accuracy compared to 74% for standard methods. When the model already starts at moderate ability, both approaches perform similarly, so the asymmetric weighting helps where it is needed and does no harm otherwise. We also provide mathematical analysis showing why upweighting hard problems is especially valuable in the early stages of learning when successes are rare.