Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models

Test-time compute (TTC) strategies have emerged as a lightweight approach to boost reasoning in large language models, but their applicability to vision-language models (VLMs) remains unclear. We present a systematic study of TTC for visual reasoning across seven open-source VLMs and six benchmarks, revisiting two paradigms: (i) feature-based scoring of chain-of-thought (CoT) traces and (ii) confidence-based aggregation via majority voting (MV). In the single-model setting, feature cues (e.g., length, pivot words) fail to improve accuracy, while MV yields only modest, CoT-dependent gains. To explain this limitation, we theoretically show that the voting method’s effectiveness depends on prediction diversity: when outputs are highly correlated, the benefit of voting vanishes. In contrast, multi-model ensembles introduce stronger diversity through architectural differences, training data, and scale, making them both more realistic and more promising for TTC. However, MV treats all models equally, leaving it vulnerable to correlated errors from weaker models. To address this, we propose Entropy-based TTC, which selects the most confident prediction based on predictive entropy. Our method reduces to MV in the single-model case but, in ensembles, leverages confidence disparities to prioritize stronger models. We prove that our method theoretically outperforms MV under mild dependence assumptions, and empirically show that it consistently surpasses both MV and the best individual model across diverse visual reasoning benchmarks. This demonstrates that smaller models can enhance, rather than hinder, larger ones when combined appropriately, unlocking ensemble gains not achievable with existing TTC strategies.

When text-based AI models are given multiple chances to answer a question and "vote" on the final result, their accuracy often improves dramatically. Our research investigates whether this strategy also works for Vision-Language Models: AI systems that must interpret both text and images. We found that simply asking a single visual AI to guess multiple times barely helps. If the AI misinterprets an image at first glance, all of its repeated guesses will usually share that exact same visual mistake. True improvement requires "diversity", which can be achieved by asking a committee of several distinctly different AI models. However, simple majority voting struggles in this setup because a group of weaker models can easily outvote a single, smarter model. To solve this, we propose Entropy-based Test-Time Compute (ETTC). Instead of treating every vote equally, ETTC evaluates the mathematical confidence of each model's prediction and selects the answer from the AI that is the most certain. Our experiments show this confidence-based method consistently outperforms basic majority voting. Crucially, ETTC allows small, inexpensive AI models to successfully assist massive, powerful models without dragging down the final results, offering a more efficient and reliable way to boost AI performance.