Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Training

Supervised fine-tuning (SFT) is computationally efficient but often yields inferior generalization compared to reinforcement learning (RL). This gap is primarily driven by RL’s use of on-policy data. We propose a framework to bridge this chasm by enabling On-Policy SFT. We first present Distribution Discriminant Theory (DDT), which explains and quantifies the alignment between data and the model-induced distribution. Leveraging DDT, we introduce two complementary techniques: (i) In-Distribution Finetuning (IDFT), a loss-level method to enhance generalization ability of SFT, and (ii) Hinted Decoding, a data-level technique that can re-align the training corpus to the model’s distribution. Extensive experiments demonstrate that our framework achieves generalization performance on par with prominent offline RL algorithms, including DPO and SimPO, while maintaining the efficiency of an SFT pipeline. The proposed framework thus offers a practical alternative in domains where RL is infeasible. We will open-source the code and data on GitHub.

AI training currently forces a choice between fast but limited methods and smart but expensive ones. Our research bridges this gap by creating "On-Policy SFT," a framework that allows models to learn with the intelligence of complex systems while maintaining the speed of simple ones. By identifying how well training data fits a model's internal logic, we can guide the AI to focus on high-quality reasoning while ignoring confusing noise that often leads to forgetting. We also introduced a technique to rewrite complex training data into a style the AI naturally understands, making difficult concepts much easier to master. Extensive testing shows that this method allows AI to outperform traditional, resource-intensive training in fields like mathematics and programming. Ultimately, this approach offers a more efficient and sustainable way to build smarter AI models without requiring massive computing power.