Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm

Cross-modal knowledge distillation (CMKD) studies how a (large) teacher model trained on one type of data (e.g., images) can guide a (smaller) student model building on another type of data (e.g., text/audio). Existing CMKD methods often require paired multi-modal data with aligned semantics, but obtaining such paired data are often costly and impractical. To mitigate this limitation, we develop a new CMKD framework for the more challenging setting where paired data are unavailable. In particular, we establish a cross-modal distributional relationship between teacher and student models which reveals two fundamental quantities governing effective distillation: feature alignment and label alignment. These quantities characterize semantic discrepancy between modalities at the levels of representation and prediction distributions, respectively. Motivated by this insight, we propose a principled framework, with theoretical guarantees, that enables effective cross-modal knowledge distillation by aligning distributions rather than individual samples. Extensive experiments across a wide range of multimodal benchmarks show that our framework is highly effective in both unpaired and paired data settings, improving significantly over prior work.

Existing cross-modal knowledge distillation methods often rely on paired data across modalities (e.g., matching images and audio), which can be expensive and difficult to obtain. We developed UCMKD, a framework that measures and reduces differences in how a teacher model and a student model represent information and make predictions, enabling knowledge transfer by aligning distributions rather than matching individual examples across modalities. This allows a student model to learn from a teacher model even when paired multimodal data are unavailable. By reducing the need for costly data collection and annotation, our approach makes cross-modal knowledge transfer more practical and broadly applicable across real-world AI systems.