Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning

Post-training of reasoning LLMs is a holistic process that typically consists of an offline SFT stage followed by an online reinforcement learning (RL) stage. However, SFT is often optimized in isolation to maximize SFT performance alone. We show that, after identical RL training, models initialized from stronger SFT checkpoints can significantly underperform those initialized from weaker ones. We propose PEAR ($\textbf{P}$olicy $\textbf{E}$valuation–inspired $\textbf{A}$lgorithm for Offline Learning Loss $\textbf{R}$eweighting), an SFT-stage method that corrects this mismatch and better prepares the model for RL. PEAR uses importance sampling to reweight the SFT loss, with three variants operating at the token, block, and sequence levels. It can be used to augment standard SFT objectives and incurs little additional training overhead once probabilities for the offline data are collected. We conduct controlled experiments on verifiable reasoning games and mathematical reasoning tasks on Qwen2.5/3 and DeepSeek-distilled models. PEAR consistently improves post-RL performance over canonical SFT, with pass@8 gains up to a 14.6% on AIME-2025. Our results suggest that PEAR is an effective step toward more holistic LLM post-training by designing and evaluating SFT with downstream RL in mind rather than in isolation.

Modern reasoning AI systems are often improved in two steps. First, they study example solutions. Then, they practice solving problems and receive feedback on whether their answers are correct. A common assumption is that making the first step as strong as possible will also produce the best final system. This paper shows that this assumption can be wrong: a model that looks better after studying examples can end up worse after the practice stage than a model that initially looked weaker. The reason is that example solutions may contain steps that are correct but unlikely to be followed by the model when it solves problems on its own. Training equally on all example steps can therefore teach habits that do not help, or even interfere with, later practice. We introduce PEAR, a simple method that gives more weight to example steps the model is likely to reuse later, and less weight to steps that are unlikely to matter during practice. Across logic puzzles and math problems, this approach consistently leads to stronger final models after practice. The broader lesson is that early training should not only make a model look good immediately; it should prepare the model to improve effectively in later stages.