Critique-Guided Distillation for Robust Reasoning via Refinement

Supervised fine-tuning with expert demonstrations often produces models that imitate outputs without internalizing the reasoning processes needed for robust generalization. While critique-based approaches show promise, training models to generate critiques directly, such as Critique Fine-Tuning (CFT), can lead to output-format drift and degradation of general capabilities. We propose $\textbf{C}$ritique-$\textbf{G}$uided $\textbf{D}$istillation (CGD), a training framework that decouples critique consumption from critique generation. During fine-tuning, the student is trained to refine flawed responses conditioned on teacher critiques. CGD treats critiques as a $\textit{training-time-only}$ supervision signal, encouraging internalization of error-aware reasoning: critiques guide learning but are absent at inference. Across five model families, CGD consistently outperforms CFT and standard distillation on mathematical reasoning benchmarks, yielding 7\% average improvements and gains of up to +15.0\% on AMC23 and +12.2\% on MATH-500. On challenging competition problems such as AIME24 and AIME25, CGD achieves substantially higher Pass@1 and stronger performance at low Pass@k, indicating improved reasoning quality per sample. Importantly, CGD preserves general instruction-following capabilities where CFT degrades significantly ($-$21.3\% on IFEval). These results position CGD as a practical and compute-efficient intermediate training paradigm for reasoning-centric tasks without introducing architectural inference-time overhead.

When we train smaller AI models by copying the outputs of larger, more capable models, they learn what to say but not why, which is like a student who memorizes answers without understanding the reasoning behind them. This limits their ability to solve new, harder problems they haven't seen before. We propose Critique-Guided Distillation (CGD), a training method where a stronger "teacher" model critiques the smaller "student" model's mistakes and explains what went wrong. The student then learns to produce corrected answers informed by this feedback. Importantly, these critiques are only used during training, i.e., once the model deployed, the student reasons on its own in a single step, with no added computational cost. Across multiple AI model families and challenging math and reasoning benchmarks, CGD consistently improves performance over existing methods while preserving the model's general abilities (such as following instructions and writing code). This offers a practical and computationally efficient path toward building AI systems that reason more robustly without requiring expensive multi-step inference or reinforcement learning.