RSAgent: Learning to Reason and Act via Multi-Turn Tool Invocations for Text-Guided Segmentation

Text-guided object segmentation requires both cross-modal reasoning and pixel grounding abilities. Most recent methods treat it as a single forward pass, where the model directly predicts pixel prompts to a segmentation model, which limits verification, refocusing and refinement when initial localization is wrong. To address this limitation, we propose RSAgent, an agentic Multimodal Large Language Model (MLLM) which interleaves reasoning and action for segmentation via multi-turn tool invocations. RSAgent queries a visual toolbox, observes feedback, and revises its pixel hypothesis using historical observations to re-localize targets and iteratively refine masks. We further build a data pipeline to synthesize multi-turn reasoning segmentation trajectories, and train RSAgent with a two-stage framework: cold-start supervised fine-tuning followed by agentic reinforcement learning with fine-grained, task-specific rewards. Extensive experiments show that RSAgent achieves a zero-shot performance of 66.5% gIoU on ReasonSeg test, improving over Seg-Zero-7B by 9%, and reaches 81.5% cIoU on RefCOCOg, demonstrating state-of-the-art performance on both in-domain and out-of-domain benchmarks.

Many useful AI systems need to find exactly which part of an image a person is talking about, such as "the cup held by the person on the left." This is difficult because the system must understand the sentence, locate the right object, and draw a precise outline around it. Existing systems often try to do this in one step: when they look at the wrong place first, they have little chance to check or fix their mistake. We introduce RSAgent, an AI system that solves the task more like a person would: it looks at the image, reasons about the request, uses visual tools to inspect or mark possible regions, checks the feedback, and then improves its answer over several rounds. To teach this behavior, we create training examples that show how to solve image segmentation through multiple steps, and then further train the system to prefer better final outlines and better intermediate actions. Our experiments show that this interactive process improves performance on several image segmentation benchmarks. The results suggest that allowing AI systems to inspect, revise, and refine their visual decisions can make them more reliable for tasks that require precise image understanding.