Dense Supervision but Sparse Guidance: A Token-Level Analysis of On-Policy Self-Distillation

On-policy self-distillation (OPSD) derives a teacher from the student’s own parameters, using privileged feedback to provide dense token-level KL supervision. We analyze this supervision on code-generation tasks and find that it often provides only sparse guidance. At the peak-disagreement position, KL frequently concentrates on tokens unrelated to code logic, such as comments and docstrings, accounting for 45 to 71% of cases. The token at this position is judged to be the primary cause of failure in only 1.1 to 8.9% of cases, although these positions can still influence what the model generates next. Allowing the teacher to continue freely from the peak-disagreement position yields correct solutions in 13.3% of cases, suggesting that the teacher’s capability is better expressed through continued generation than through a single-token score. At the same time, the low success rate reveals a limitation inherent in parameter-shared teachers. More strikingly, substituting only the teacher-preferred token and allowing the student to continue raises success to 9.9%, recovering 71% of the teacher’s free continuations. Our findings explain why SDPO can outperform GRPO: its benefit may come less from reliable local correction and more from concentrated guidance that influences the subsequent generation trajectory. More broadly, local corrective quality and optimization usefulness can diverge in token-level supervision.