The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works

Knowledge distillation (KD) transfers knowledge from a large teacher model to a smaller student. In language modeling, the student is trained either on tokens sampled from the teacher (\textbf{hard labels}) or the teacher’s full next-token distribution (\textbf{soft labels}). Despite soft labels appear strictly richer, we find that mixing hard and soft labels consistently yields better results. Crucially, we show that this gain cannot be explained by closer teacher matching during training. Instead, it comes from reduced exposure bias---the mismatch between training and inference distributions. To explain this phenomenon, we introduce the Bridge–Garden Decomposition theory, which categorizes generation steps into two types: \textit{Bridges}, where the next token must be \textit{exact}, and \textit{Gardens}, where it can be \textit{flexible}. We show that hard-only KD excels in Bridges by avoiding risky deviations, while soft-only KD preserves diversity in Gardens. A hybrid strategy handles both cases and, as a result, reduces exposure bias across the sequence. Guided by this theory, we develop a family of Bridge--Garden hybrid supervision methods that adaptively balance hard and soft labels. Across seven teacher--student pairs (including Qwen, Llama, Gemma, and DeepSeek) and benchmarks in reasoning and coding, our approach outperforms divergence-based and on-policy KD baselines while reducing training cost by \textbf{9.7$\times$}, enabling efficient model compression.

Large language models are useful, but they are expensive to run. A common way to reduce this cost is to train a smaller model to imitate a larger one. This paper studies a simple question in that process: when the larger model teaches the smaller one, should it provide only its chosen answer, or should it also provide its full range of possible answers? At first, the full range of possible answers seems strictly better, because it contains more information. We find that this is not always true. Some parts of generation are narrow: one wrong word, symbol, or code token can derail the whole answer. Other parts are flexible: several word choices can be equally acceptable. The first kind benefits from clearer, more decisive supervision, while the second benefits from preserving alternatives. Based on this observation, we propose methods that mix these two kinds of teaching signals. Across reasoning and coding tasks, this gives smaller models better performance while keeping training much cheaper than more expensive alternatives.