What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoom

Vision tool-use reinforcement learning (RL) can equip vision--language models with visual operators such as crop-and-zoom and achieves strong performance gains, yet it remains unclear whether these gains are driven by improvements in tool use or evolving intrinsic capabilities. We introduce \textbf{MED} (Measure--Explain--Diagnose), a coarse-to-fine framework that disentangles intrinsic capability changes from tool-induced effects, decomposes the tool-induced performance difference into gain and harm terms, and probes the mechanisms driving their evolution. Across checkpoint-level analyses on two VLMs with different tool priors and six benchmarks, we find that improvements are dominated by intrinsic learning, while tool-use RL mainly reduces tool-induced harm (e.g., fewer call-induced errors and weaker tool schema interference) and yields limited progress in tool-based correction of intrinsic failures.Overall, current vision tool-use RL learns to coexist safely with tools rather than master them.

Many AI systems that answer questions about images look at the whole image once and then reason from that fixed view. Newer systems can use visual tools, such as zooming into part of an image, and are trained with rewards for correct answers. It is tempting to assume that better scores mean they have learned to use the tool well. Our paper tests this assumption. We compare each model’s performance with and without the tool throughout training, and separate improvements that come from general visual understanding from improvements caused by tool use itself. We find that much of the progress comes from the model becoming better even without tools. When tools help, training mainly makes tool use less harmful, for example by reducing bad zooms or distractions from the tool instructions, rather than consistently solving the hardest cases that the model could not answer without tools. This matters because future AI systems should not only be rewarded for calling tools more often; they should be trained and evaluated to use tools selectively, when the tool actually helps.