Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge

Large Language Models (LLMs) possess broad conceptual knowledge acquired through large-scale text pretraining, yet their potential to supervise models in other modalities remains underexplored. In this work, we propose LaViD—Language-to-Visual Knowledge Distillation—a simple and effective framework for transferring high-level semantic knowledge from a language-only teacher to a vision-only student model. Instead of relying on paired multimodal data, LaViD elicits conceptual signals from an LLM by prompting it to generate multiple-choice questions (MCQs) that probe semantic distinctions between visual classes. Each class is mapped to a soft label distribution over these MCQs, forming a rich conceptual signature that guides the student through an auxiliary distillation loss. Notably, despite using a language-only teacher without access to image data, LaViD consistently outperforms recent methods like MaKD that distill from vision-language models across multiple fine-grained benchmarks. It also achieves competitive or superior performance compared to state-of-the-art visual distillation methods such as DKD and MLKD, with further gains when combined with logit standardization. On the Waterbirds dataset, LaViD substantially improves worst-group accuracy, demonstrating enhanced robustness to spurious correlations with distillation. Code is available at \url{https://github.com/lliangthomas/lavid}.

Many image recognition systems learn mainly from labeled images. This can work well, but it may also cause models to focus on accidental visual patterns, such as background, rather than on the object itself. In this paper, we study whether knowledge from a language model can help an image model learn more meaningful visual concepts. Our method asks a language model to create simple multiple-choice questions about the differences between visual categories, such as color, shape, or other distinctive traits. The language model then provides soft guidance about how each category relates to these questions. An image model is trained not only to predict the correct category, but also to match this concept-level guidance. Across several fine-grained image recognition tasks, this approach improves performance over strong baselines, even though the language model never sees the training images. It also helps the image model become less dependent on misleading background cues in a biased dataset. These results suggest that language models can provide useful conceptual knowledge for training better visual models.