Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs

Reinforcement learning (RL) is widely used to improve large language models (LLMs) on reasoning tasks, and asynchronous RL training is attractive because it increases end-to-end throughput. However, for widely adopted critic-free policy-gradient methods such as REINFORCE and GRPO, high asynchrony makes the policy-gradient estimator markedly higher variance: stale off-policy rollouts induce heavy-tailed importance ratios, causing a small fraction of samples to dominate each update. This amplification makes gradients noisy and learning unstable relative to matched on-policy training. Across math and reasoning benchmarks, we find collapse is preceded by sharp drops in effective sample size (ESS) and unstable gradient norms. Motivated by this diagnosis, we propose Variance Controlled Policy Optimization (VCPO), a drop-in stabilization method for REINFORCE/GRPO-style algorithms that (i) rescales learning rate according to effective sample size to dampen unreliable updates, and (ii) applies a closed-form minimum-variance baseline for the off-policy setting, avoiding an auxiliary value model and adding minimal overhead. Empirically, VCPO substantially improves robustness in highly asynchronous regimes across models sizes and tasks, reducing long-context, multi-turn training compute by 1.96×. Overall, our results demonstrate explicitly controlling policy-gradient variance is key to making asynchronous RL reliable at scale.