Timezone: »

 
Poster
Text-To-Concept (and Back) via Cross-Model Alignment
Mazda Moayeri · Keivan Rezaei · Maziar Sanjabi · Soheil Feizi

Tue Jul 25 02:00 PM -- 04:30 PM (PDT) @ Exhibit Hall 1 #735
in Poster Session 1 »

We observe that the mapping between an image's representation in one model to its representation in another can be learned surprisingly well with just a linear layer, even across diverse models. Building on this observation, we propose text-to-concept, where features from a fixed pretrained model are aligned linearly to the CLIP space, so that text embeddings from CLIP's text encoder become directly comparable to the aligned features. With text-to-concept, we convert fixed off-the-shelf vision encoders to surprisingly strong zero-shot classifiers for free, with accuracy at times even surpassing that of CLIP, despite being much smaller models and trained on a small fraction of the data compared to CLIP. We show other immediate use-cases of text-to-concept, like building concept bottleneck models with no concept supervision, diagnosing distribution shifts in terms of human concepts, and retrieving images satisfying a set of text-based constraints. Lastly, we demonstrate the feasibility of concept-to-text, where vectors in a model's feature space are decoded by first aligning to the CLIP before being fed to a GPT-based generative model. Our work suggests existing deep models, with presumably diverse architectures and training, represent input samples relatively similarly, and a two-way communication across model representation spaces and to humans (through language) is viable.

[ PDF]  [ Poster] 

Author Information

Mazda Moayeri (University of Maryland)
Keivan Rezaei (University of Maryland)
Maziar Sanjabi (Meta AI)
Soheil Feizi (University of Maryland)

More from the Same Authors