Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling

Vision-Language Models (VLMs) excel at tasks like zero-shot classification and cross-modal retrieval by mapping images and text to a shared space, but this requires expensive end-to-end training with massive paired datasets. Current post-hoc alignment methods reduce computational costs by connecting pretrained encoders through lightweight mappings, yet still demand substantial paired data. In this work, we investigate the potential of repurposing the classification heads of pretrained vision models as semantic prototypes. The recycling of these weights, typically discarded after pretraining, unlocks two distinct capabilities: it enables zero-shot alignment by using weights as semantic anchors, and serves as a robust data augmentation strategy by mixing these prototypes with real image-text pairs. We demonstrate that integrating our approach with several state-of-the-art post-hoc alignment techniques consistently boosts accuracy in cross-modal retrieval, zero- and few-shot classification tasks.

Modern AI systems can connect images and text, such as finding pictures from a written description or recognizing a new category from its name. Building these systems usually requires huge collections of matched images and captions, which are expensive to collect and may not exist for specialized fields. Our paper shows that standard image classifiers already contain a useful hidden bridge to language. The final classification layer stores one vector for each class it was trained to recognize, and these vectors behave like compact descriptions of visual concepts. Instead of throwing this layer away after (pre)training, we reuse its class vectors and pair them with the corresponding class names to connect a pretrained image model with a text model. This gives a vision-only model some image-text abilities without needing image-caption pairs for that connection step. When a small number of image-caption pairs is available, adding these recycled class vectors improves existing methods, especially in low-data settings. We test the idea on image classification, few-shot learning, and finding matching images and captions, and find that it consistently helps. The broader message is that supervised pretrained image models may contain reusable semantic knowledge that can make vision-language systems cheaper and easier to adapt.