A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions

Despite the success of knowledge distillation (KD) in Large Language Models (LLMs), the underlying mechanism behind its efficacy remains unclear. In this paper, we propose a unified approach to explore the common mechanism of various KD methods using interactions. Specifically, we decompose the output score of the LLM into the sum of numerous interactions. Each interaction represents a nonlinear relationship involving a set of input variables (e.g., words). Based on the decomposed interactions, we discover that the common mechanism underlying various KD methods is the sparsification of interactions, i.e., student models retain fewer interactions for inference while suppressing other interactions to zero effects. Furthermore, we discover that the performance variance across different KD methods arises from their capabilities in handling complex interactions. A KD method typically yields better performance if it enables the student model to achieve higher sparsity of complex interactions. Motivated by these insights, we propose a plug-and-play loss function called Complex Interaction Penalty (CIP) to explicitly enforce the sparsity of complex interactions during the distillation process. Extensive experiments demonstrate that integrating CIP consistently improves the performance of diverse KD methods on both in-domain and out-of-distribution benchmarks.

Large language models (LLMs) are powerful but require massive amounts of computing power. To fix this issue, researchers use a technique called Knowledge Distillation (KD) to compress a massive "teacher" model into a smaller "student" model. However, the exact reason why KD works so well has remained a mystery. We explored the mechanism of KD based on interactions, which represent the relationships between different input words. We discovered that KD acts like a filter: the student model learns important interactions from the teacher while discarding unimportant interactions. Based on this insight, we designed a new loss function that can be integrated with various KD methods. It consistently improves the performance of various KD methods. Our work improves how we build efficient LLMs and opens the black box to explain how KD actually works.