WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning

Group Relative Policy Optimization (GRPO) is effective for training language models on complex reasoning. However, since the objective is defined relative to a group of sampled trajectories, extended deliberation can create more chances to realize relative gains, leading to inefficient reasoning and overthinking, and complicating the trade-off between correctness and rollout efficiency. Controlling this behavior is difficult in practice, considering (i) Length penalties are hard to calibrate because longer rollouts may reflect harder problems that require longer reasoning, penalizing tokens risks truncating useful reasoning along with redundant continuation; and (ii) supervision that directly indicates when to continue or stop is typically unavailable beyond final answer correctness. We propose Weakly Supervised GRPO (WS-GRPO), which improves rollout efficiency by converting terminal rewards into correctness-aware guidance over partial trajectories. Unlike global length penalties that are hard to calibrate, WS-GRPO trains a preference model from outcome-only correctness to produce prefix-level signals that indicate when additional continuation is beneficial. Thus, WS-GRPO supplies outcome-derived continue/stop guidance, reducing redundant deliberation while maintaining accuracy. We provide theoretical results and empirically show on reasoning benchmarks that WS-GRPO substantially reduces rollout length while remaining competitive with GRPO baselines.

Modern AI reasoning models solve problems by generating step-by-step "chains of thought," but they often keep reasoning long after they have effectively reached the answer. A common training method, GRPO, rewards trajectories based only on final-answer correctness, which can inadvertently push models toward longer chains since these have more chances to land on the right answer. We introduce WS-GRPO, which learns from the same final-answer correctness signals to identify when continuing to reason is genuinely useful, without any expensive step-by-step human annotation. Across four reasoning benchmarks and four model families, WS-GRPO produces reasoning chains up to 93% shorter and 83% fewer steps while remaining competitive in accuracy.