How to Fine-Tune a Reasoning Model? A Teacher–Student Cooperation Framework to Synthesize Student-Consistent SFT Data

A widely adopted strategy for model enhancement is to use synthetic data generated by a stronger model for supervised fine-tuning (SFT). However, for emerging reasoning models like Qwen3-8B, this approach often fails to improve reasoning capabilities and can even lead to a substantial drop in performance. In this work, we identify substantial stylistic divergence between teacher generated data and the distribution of student as a major factor impacting SFT. To bridge this gap, we propose a Teacher–Student Cooperation Data Synthesis framework (TESSY), which interleaves teacher and student models to alternately generate style and non-style tokens. Consequently, TESSY produces synthetic sequences that inherit the advanced reasoning capabilities of the teacher while maintaining stylistic consistency with the distribution of the student. In experiments on code generation using GPT-OSS-120B as the teacher, fine-tuning Qwen3-8B on teacher-generated data leads to performance drops of 3.25% on LiveCodeBench-Pro and 10.02% on OJBench, whereas TESSY achieves improvements of 11.25% and 6.68%.

Training smaller student models on data synthesized by stronger teachers is a standard practice for capability enhancement. However, this approach falters when applied to large language models with deep thinking abilities, often leading to a decline in student capability. We found that a key reason for this is the stylistic gap between teachers and students. Due to their distinct linguistic habits, forcing a student to learn from a teacher’s data causes it to mimic the teacher's "accent" rather than internalizing core "knowledge," resulting in superficial imitation. To address this, we propose the Teacher–Student Cooperation Data Synthesis framework. Instead of relying on a single model, this framework enables the student and teacher models to perform distinct roles by synthesizing different parts of each data entry. This collaborative process ensures the final training data is both easy for the student to understand and rich in the teacher's expert knowledge.