Diversity Over Frequency: Rethinking Tool Use in Visual Chain-of-Thought Agents

Visual agents employ external visual tools within visual chains of thought to incorporate fine-grained evidence. While prior work has mainly studied these tools in visual search tasks, their role in more complex visual reasoning remains underexplored. In this paper, we move beyond simple visual search tasks to investigate more challenging tasks, including 3D spatial reasoning and medical visual question answering, where agents must integrate tool-acquired local evidence with the global context. We identify a \emph{tool-use collapse phenomenon}: models progressively stop using tools while still achieving higher task accuracy. Moreover, we observe a clear asymmetry: (i) completely eliminating tool use degrades performance, whereas (ii) incentivizing tool use yields only marginal gains despite substantially increasing usage. We find that vanilla training and tool-use encouragement both reduce rollout diversity, explaining why higher tool use does not yield stronger reasoning performance. Motivated by these findings, we add an entropy regularization term to encourage diverse rollout exploration, achieving the best performance despite gradually declining tool usage. Overall, our findings suggest a training-time view of tools as scaffolding, where broader exploration over language generation and visual tool invocation improves reasoning despite tool-use collapse. Project page: https://scaffolded-exploration.github.io

Many AI systems that answer questions about images can use helper tools, for example zooming into a region, to inspect details before answering. We studied what happens when such systems are trained on harder visual reasoning tasks, including judging 3D spatial relationships in everyday photos and answering questions about medical images. Surprisingly, standard training often made the system stop using its tools, even while its answers improved. But removing the tools from the start made the system worse, and simply rewarding it for using tools made it call them almost all the time without much extra benefit. We found that what mattered was not how often the tools were used, but whether early training exposed the system to a diverse set of thoughts and image regions. We therefore added a training signal that keeps the system exploring different reasoning paths and visual evidence. This led to the best performance, even though tool use still faded later. Our results suggest that visual tools can act like training wheels. They help AI systems learn better ways to reason, even if they are not needed much at test time. This could make future visual AI agents more accurate and efficient, because they may rely less on costly tool calls while keeping the benefits of having explored them during training.