DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agents

Agentic multimodal models have garnered significant attention for their ability to leverage external tools to tackle complex tasks. However, it is observed that such agents often meet premature interaction collapse, caused by two primary reasons: 1) the terminal reward often appending on the last token prevents the advantage from distinguishing trajectories with exploratory behavior; 2) excessively redundant context hinders the agent from absorbing useful feedback. To address these issues, we propose the Deepening Reasoning MMSearchAgent, the framework leverages the structural proximity to derive advantage signals from the whole rollout trajectories in an entire batch, such that trajectories of different lengths are further encouraged to be generated, even when containing the same correct answer. Additionally, differentiated gaussian rewards are employed to dynamically calibrate interaction tolerance, thereby ensuring information reliability and reducing redundancy. To support multi-turn interaction training, we have constructed a multi-step deep-reasoning dataset including 3602 high-quality QA pairs with at least 3 reasoning steps. Extensive experiments demonstrate that our method achieves state-of-the-art performance, outperforming the MMSearch-R1 by 8.4\% on FVQA-test.

(1) Advanced AI assistants that use external tools to solve complex, multi-step problems are very promising, but they often fail by giving up too early. This premature collapse happens because the AI gets easily overwhelmed by too much repetitive information and because traditional training methods only reward the final answer, failing to recognize the step-by-step exploratory process needed to get there. (2) To tackle this, we developed the Deepening Reasoning MMSearchAgent. Our approach changes how the AI is trained: we utilize rewards and response lengths to further distinguish advantage diversity among different trajectories. We also implemented a dynamic system to filter out redundant information, helping the AI focus on reliable feedback. (3) This improved training method helps AI assistants stay focused and reason deeper. Our results show it significantly outperforms existing methods, making these systems much better at solving intricate problems that require sustained investigation.