Entropy-aware Span-Constrained Optimal Transport for Robust Cross-Tokenizer Knowledge Distillation

Existing Cross-Tokenizer Knowledge Distillation (CTKD) methods can fail to outperform simple supervised fine-tuning when vocabulary overlap is low due to severe alignment noise. We identify this phenomenon as the ``Low-Overlap negative transfer regime''. To overcome this, we propose Entropy-aware Span-Constrained Optimal Transport (E-SCOT), a robust framework that treats distillation as a sparse transport problem built upon a vocabulary-agnostic ground metric. Unlike prior OT approaches that incur quadratic costs via dense sequence-level optimization, E-SCOT employs span-anchored lexical alignment to construct a deterministic, locality-preserving support set in linear time with respect to sequence length. Furthermore, we introduce R\'enyi-entropy adaptive reweighting to dynamically concentrate the distillation budget on informative positions exhibiting significant uncertainty-profile gaps. Extensive experiments demonstrate that E-SCOT achieves state-of-the-art performance across diverse model families, effectively eliminating negative transfer even in challenging low-overlap scenarios.

Large language models are powerful but often too costly to deploy in practical settings, so researchers train smaller models to imitate larger ones. This imitation process becomes difficult when the two models split text into different pieces, as is common across model families. In such cases, existing methods can introduce noisy supervision and may even make the smaller model perform worse than simply training it on the task data. This paper proposes E-SCOT, a method for helping a smaller model learn from a larger model when their ways of splitting text differ. Instead of forcing words or subwords from two different vocabularies to match directly, E-SCOT uses the original text as a shared reference and matches nearby text spans. It also puts more emphasis on parts of the text where the smaller model's uncertainty differs most from the larger model's. Our experiments show that E-SCOT helps smaller models learn more reliably from larger models across diverse model families, especially when the two models split text in very different ways.