Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers

In-context learning (ICL) enables multimodal large language models (MLLMs) to classify images from a few labeled examples. However, how these models leverage the provided context remains poorly understood. While Chain-of-Thought prompting is widely adopted, recent work suggests it may not faithfully reflect the models’ internal reasoning processes. In this paper, we systematically evaluate the concept-based explainability of frozen MLLMs under few-shot ICL across five conditions of increasing formal rigor, ranging from baseline classification to Description Logics (DL) axiom generation. We assess four state-of-the-art MLLMs using an independent LLM-as-a-judge evaluation pipeline and show that explaining is genuinely harder than predicting. Notably, enforcing formally structured, concept-based explanations leads to a monotonic decrease in predictive accuracy (from 93.8% to 90.1%), challenging the common assumption that explicit reasoning universally improves performance. At the same time, when models successfully identify class-discriminative visual features, explanation quality strongly correlates with correct predictions. Overall, our results indicate that while MLLMs are highly effective at visual classification, they lack the instruction tuning necessary to produce formal, machine-verifiable explanations.