OnePO: Direct One-stage Policy Optimization for SFT-free Domain Adaptation

Domain adaptation typically follows a two-stage pipeline: Supervised Fine-Tuning (SFT) then Reinforcement Learning (RL). However, does RL necessarily require a pre-SFT phase for domain adaptation? SFT confines the model to an imitation distribution, limiting RL exploration, while the two-stage transition causes capability regression and extra engineering. We propose One-stage Policy Optimization (OnePO), an SFT-free paradigm that adapts pretrained LLMs to target domains in a single RL stage. OnePO uses teacher outputs as transient guidance to overcome the slow convergence of pure RL, while avoiding two failures of naive teacher-output integration: gradient starvation for low-probability teacher tokens and distribution anchoring from persistent teacher signals. It introduces two mechanisms: (1) Adaptive Objective Evolution, reshaping the RL objective for rapid absorption of teacher-provided knowledge; and (2) Teacher Retirement, automatically discarding teacher outputs once the model surpasses them. On medical adaptation, OnePO achieves 67.2 on HealthBench with only 20K training samples, outperforming SFT+RL by +2.7 and pure RL by +7.4 points. Scaling the same recipe produces HuatuoGPT-3, an open-source medical LLM series whose 32B variant reaches 70.3 on HealthBench. Additional writing and legal-domain experiments show that OnePO extends beyond medicine. Models and code are available at https://github.com/FreedomIntelligence/HuatuoGPT-3.

Large language models are increasingly used in specialized fields such as medicine, but adapting a general model into a reliable domain expert is still difficult. A common approach first teaches the model to imitate expert answers and then lets it improve through feedback. This two-step process can be costly and may limit the model by making it copy the teacher too closely. We propose OnePO, a simpler one-step training method. Instead of first forcing the model to imitate expert answers, OnePO uses teacher answers only as temporary guidance. Early in training, these answers help the model learn useful domain knowledge more quickly. Later, the method automatically reduces reliance on the teacher so the model can continue improving on its own. In medical experiments, OnePO trained strong models using only 20,000 examples and outperformed both the standard two-step approach and training without teacher guidance. We also used it to build HuatuoGPT-3, an open-source family of medical language models, and found that the same idea can help in writing and legal tasks as well.