Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling

Existing LLMs-post-training techniques are broadly categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). Each paradigm presents a distinct trade-off: (1) SFT excels at mimicking demonstration data, but can lead to problematic generalization as a form of behavior cloning. (2) Conversely, RFT can significantly enhance a model's performance but is prone to learning unexpected behaviors, and its performance is sensitive to the initial policy. In this paper, we propose a unified view of these methods and introduce Prefix-RFT, a hybrid approach that synergizes learning from both demonstration and exploration. Using mathematical reasoning problems as a test bed, we empirically demonstrate that Prefix-RFT is simple yet effective. Not only does it surpass the performance of standalone SFT and RFT, but it also outperforms parallel mixed-policy RFT methods. Our analysis highlights the complementary nature of SFT and RFT, validating that Prefix-RFT effectively harmonizes them. Further ablation studies confirm the method's robustness to variations in the quality and quantity of demonstration data.

Language models are improved after pretraining by either cloning high-quality example answers (Supervised Fine-Tuning, SFT) or by learning through trial and error with rewards (Reinforcement Fine-Tuning, RFT). SFT can make a model too rigid, while trial-and-error learning can struggle when the model does not know how to solve a hard problem. Our work studies how to combine the strengths of both approaches. We propose Prefix-RFT, a training method that gives the model only part of a good solution and then asks it to finish the rest. The full answer is then judged by whether it solves the problem, so the model learns from helpful examples without being forced to copy every step. This creates a balance between expert guidance and independent exploration. We test this idea on challenging math and code tasks. Prefix-RFT improves performance over standard imitation-based and reward-based training, as well as several hybrid alternatives. This matters because it offers a simple way to train stronger reasoning models while using demonstrations more selectively and efficiently.