On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse

Group Relative Policy Optimization (GRPO) eliminates the learned critic in PPO by using the mean reward of grouped rollouts as a baseline. We provide a rigorous derivation of GRPO from first principles of the policy gradient theorem, re- vealing a fundamental credit assignment failure: under output-only reward, every token in a roll- out receives identical advantage, collapsing token- level credit to a single scalar. We prove this in- duces gradient sparsity that intensifies over train- ing, and demonstrate empirically via SVD analy- sis of GRPO gradients on Nemotron-4B/GSM8K that the gradient matrix has effective rank ≈ 2 regardless of group size R ∈ {2, 4, 8}. Our results characterize when GRPO’s simplicity is theoretically justified and identify the credit assignment bottleneck as the key limitation for multi-step reasoning.