Cheap When Confident: The Per-Model Limits of Selective Answering in Multimodal Geometry QA

An efficient multimodal QA system cannot afford to run an expensive pipeline on every question. A standard lever is selective answering: answer the questions the model is confident in and defer the rest, to abstention, to retrieval, or to a stronger and costlier model. This lever is only as good as the model’s confidence is calibrated, and the calibration of vision-language models on visual mathematical reasoning is poorly characterized. We study confidence-gated selective answering on diagram-based plane-geometry QA from MathVerse with two vision-language models of very different cost, Claude Haiku and Pixtral-12B, reporting the full risk–coverage trade-off alongside accuracy, calibration (ECE), confidence on wrong answers, and abstention. Two findings bear on efficiency. First, whether confidence is usable as a routing signal is a property of the model, not a knob: at nearly identical accuracy (0.67– 0.69), confidence on wrong answers is 0.70 for Claude and 0.94 for Pixtral, so confidence-gated answering buys back accuracy on Claude, whose selective accuracy climbs from 0.72 to 0.80 at 60% coverage and 0.86 at 40%, but not on Pixtral, whose confidence is so concentrated that 63% of its answers share the single top confidence value: thresholding cannot reduce coverage below 63%, and selective accuracy never exceeds 0.75. Second, a lightweight, no-training, in-context selfimprovement loop driven by verifiable feedback does not change this. Across four rounds, steering the loop by the model’s confidence versus by its verified correctness leaves both accuracy and calibration statistically unchanged (paired 95% intervals on both differences span zero, McNemar p > 0.3), so the cheap inference-time route to a more routable model does not work here. The practical implication is that confidence-gated efficiency must be validated per model before deployment, since a verifiable reward does not buy a routable agent. Results are scoped to a small setting (N=40, multiple choice).