Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning

Tool-augmented reasoning has emerged as a promising direction for enhancing the reasoning capabilities of multimodal large language models (MLLMs). However, existing studies mainly focus on enabling models to perform tool invocation, while neglecting the necessity of invoking tools. We argue that tool usage is not always beneficial, as redundant or inappropriate invocations largely increase reasoning overhead and even mislead model predictions. To address this issue, we introduce AutoTool, a model that adaptively decides whether to invoke tools according to the characteristics of each query. Within a reinforcement learning framework, we design an explicit dual-mode reasoning strategy with mode-specific reward functions to guide the model toward producing accurate responses. Moreover, to prevent premature bias toward a single reasoning mode, AutoTool jointly explores and balances tool-assisted and text-centric reasoning throughout training, and promotes free exploration in later stages. Extensive experiments demonstrate that AutoTool exhibits outstanding performance and high efficiency, yielding a 21.8\% accuracy gain on V* benchmark compared to the base model, and a 44.9\% improvement in efficiency over existing tool-augmented methods on POPE benchmark. Code is available at https://github.com/MQinghe/AutoTool.

Modern Multimodal Artificial Intelligence (AI) models have significantly improved their capability to process complex visual data by leveraging external tools, such as zooming mechanisms to inspect fine-grained image details. However, current tool-augmented models typically suffer from a critical limitation: they invoke external tools indiscriminately for every query, even when a problem is straightforward enough to be solved directly. This inefficient behavior unnecessarily increases computational latency and can introduce irrelevant noise that impairs the model's final predictions. To address this challenge, we developed AutoTool, a framework that enables AI models to adaptively determine whether tool invocation is genuinely required for a given task. Utilizing a reinforcement learning framework, we incorporate mode-specific reward functions that incentivize the model to choose the most appropriate reasoning pathway while penalizing redundant tool usage. Our experiments demonstrate that AutoTool successfully balances performance and efficiency.