What Do Chart Question Answering Benchmarks Measure? Task-Specific Visual Dependence in Efficient Multimodal QA

Aggregate accuracy on multimodal benchmarks is often interpreted as evidence that vision-language models (VLMs) use visual information effectively. However, aggregate scores can obscure whether models depend on visual evidence in the same way across task types. We study this question in chart question answering using counterfactual modality interventions on ChartQA and a PlotQA-derived evaluation subset. We compare matched-image, no-image, and distractor-image conditions while partitioning questions into lookup, compositional, and yes/no categories. Across multiple VLMs, image removal produces the largest and most consistent degradation for lookup questions, substantial but smaller drops for compositional questions, and more variable effects for yes/no questions. Distractor-image results further suggest that lookup and compositional performance depend on matched visual evidence rather than generic chart-like input. On a 100-example ChartQA subset, removing image input reduced Qwen2-VL-2B inference latency by 30.6\%, suggesting that selective visual computation may improve multimodal inference efficiency. These findings motivate task-specific visual-dependence reporting both as an evaluation diagnostic and as a possible routing signal for efficient multimodal QA systems.