Making Expert Reasoning Learnable with Self-Distillation

Improving the reasoning capabilities of large language models (LLMs) typically relies either on the model's ability to sample a correct solution to be reinforced or the existence of a stronger model able to solve the problem. However, many difficult problems remain intractable for even current frontier models, preventing the extraction of valid training signals. A promising alternative is to leverage high-quality expert human solutions, yet naive imitation of this data fails because it is fundamentally out-of-distribution: expert solutions are typically didactic, containing implicit reasoning gaps intended for human readers rather than computational models. Furthermore, high-quality expert solutions are expensive, necessitating generalizable sample-efficient training methods. We propose Distribution Aligned Imitation Learning (DAIL), a two-step self-distillation method that bridges the distributional gap by first transforming expert solutions into detailed, in-distribution reasoning traces and then applying a contrastive objective to focus learning on expert insights and methodologies. We find that DAIL can leverage fewer than 1000 high-quality expert solutions to achieve up to 31% pass@128 gains on Qwen2.5-Instruct and Qwen3, double reasoning efficiency, and enable out-of-domain generalization.

Training artificial intelligence (AI) models to solve complex reasoning problems is challenging. Usually, models learn from success or failures through reinforcement learning, but for the hardest problems, even the best AI models currently struggle. While for these problems, we could use solutions written by human experts, models have trouble learning from them because humans often skip obvious steps in their explanations, leaving gaps that would confuse models if they were trained on them. Furthermore, gathering this expert data is very expensive. To address this issue, we developed a new training method called Distribution Aligned Imitation Learning (DAIL). First, DAIL acts like a translator, turning the human expert's high-level explanations into the highly detailed and complete step-by-step instructions that a model understands and can learn from easily. Then, it uses a specialized learning technique to help the AI focus specifically on the translated solution's core insights and problem-solving methods. Our experiments show that DAIL is highly efficient: using fewer than 1,000 human expert examples, we can significantly improve the reasoning abilities of existing AI models. Not only did the model become twice as efficient at solving problems, but it also learned to successfully apply these reasoning skills to completely new problem domains.