Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO

We identify a new dimension for enhancing rollout diversity in Group Relative Policy Optimization (GRPO) for LLMs. While GRPO relies on diverse rollouts, prevailing strategies primarily increase diversity by injecting more token-level randomness, which may introduce step-wise noise and leads to incoherent trajectories. We uncover that smaller models within the same model family inherently exhibit higher policy-level diversity, indicated by their superior pass@k relative to larger counterparts as sample counts increase. Unlike token-level noise, this diversity is temporally correlated, preserves logical consistency, and provides structured exploration signals for gradient estimation. We thus propose S2L-PO (Small-to-Large Policy Optimization), a framework that leverages fixed small models as natural explorers to train larger models. To balance exploration and exploitation, we design a progressive annealing strategy that transitions from offline small-model rollouts to the large learner’s own sampling. This shift elegantly avoids mid-training performance drops caused by the small model's capacity limits, achieving faster convergence and unlocking a higher performance ceiling. S2L-PO improves accuracy on diverse mathematical reasoning benchmarks (eg., +8.8\% on AIME 24 using a 1.7B explorer to guide the 8B model) while reducing rollout compute. The code will be made available.

Training AI systems to reason well—like solving math competition problems—typically involves having the AI practice by generating many candidate solutions and learning from the ones that work. A key challenge is making sure the AI tries sufficiently diverse approaches rather than repeatedly attempting minor variations of the same strategy. A common fix is to add randomness to the AI's outputs, but this turns out to be a poor substitute for genuine diversity: random noise makes individual steps unpredictable without encouraging the model to pursue fundamentally different solution paths. We observed a surprising phenomenon: smaller AI models from the same family naturally produce a wider variety of distinct strategies than their larger counterparts, even when given the same problems. This happens because compression forces a smaller model to develop different, often more varied, problem-solving habits. We exploit this property in a simple way. During training of a large model, we mix in solution attempts generated by a frozen, smaller sibling model alongside the large model's own attempts. The large model then learns from this richer, more varied pool of examples. As training progresses, we gradually phase out the smaller model's contributions so that the large model ultimately trains on its own outputs—exactly as in standard practice. This approach, which we call S2L-PO (Small-to-Large Policy Optimization), consistently improves reasoning performance. On challenging mathematics benchmarks, it yields substantial accuracy gains while actually reducing the computational cost of the training process. Importantly, the benefits extend beyond math: the large model also improves on commonsense reasoning tasks it was never explicitly trained on, suggesting that richer exploration during training leads to more broadly capable models.