Zero-Shot Rankability: Revealing Latent Ordinal Structure in Multimodal Large Language Models via Language

Recent work shows that vision encoders capture ordinal attributes along linear axes, which can be recovered from as few as two labeled images. However, in the zero-shot setting, the text-driven rank axis for Vision-Language Models (VLMs) like CLIP remains suboptimal. In this work, we study the embeddings of Multimodal LLMs (MLLMs). We hypothesize that MLLMs can overcome this limitation due to three potential advantages: their inherent ordinal understanding, capacity for conditional embeddings, and a small cross-modal gap. We show that MLLMs are rankable using only text prompts. Experiments demonstrate that a text-driven rank axis for MLLM embeddings achieves 90\% of the performance of the supervised linear rank axis, significantly outperforming the 61\% observed in VLM embeddings. We validate that this capability stems from MLLMs' conditional embeddings and a smaller modality gap than VLMs. Furthermore, we demonstrate that this property generalizes to the audio domain. Our findings suggest that language provides a direct interface for probing latent ordinal structures in MLLMs. Code and prompts are available at https://github.com/kaist-ami/prompt-probing

Many valuable image assessments are inherently graded rather than categorical, including perceived age, scene crowding, aesthetic appeal, and recency. Although modern AI systems seem to internally represent these graded attributes, accessing this information typically requires extensive labeled data. Previous efforts to infer such orderings solely from textual attribute descriptions have performed poorly on widely used image-text models such as CLIP. This study examines Multimodal Large Language Models, a recent class of AI systems that integrate visual and linguistic components. Results indicate that a simple natural-language instruction enables recovery of the underlying ordering of images by age, crowd density, photo quality, or recency. This capability is attributed to two key properties of the newer models: the adaptability of their internal representations to specific queries and a closer alignment between visual and linguistic encoding. Natural language now offers a direct means to access how these AI systems internally rank and compare attributes, and this approach is also applicable to audio. This advancement provides researchers with a more straightforward method to audit model understanding of graded properties and offers practitioners a label-free tool for scoring and ranking tasks.