Ophiuchus: Incentivizing Tool-augmented ''Think with Images'' for Joint Medical Segmentation, Understanding and Reasoning

Recent medical MLLMs have made significant progress in generating step by step textual reasoning chains. However, they still struggle with complex clinical tasks that necessitate dynamic and iterative focusing on fine-grained visual regions. To close this gap, we introduce Ophiuchus, a versatile, tool-augmented framework that equips an MLLM to (i) decide when fine-grained visual evidence is needed, (ii) determine where to probe and ground within the medical image, and (iii) seamlessly weave the relevant sub-image content back into an interleaved, multimodal chain of thought for precise segmentation and diagnosis. Ophiuchus moves beyond mere tool-calling by tightly fusing the MLLM’s inherent grounding and reasoning capabilities with external tools, enabling more accurate and trustworthy decisions. The core of our method is a three-stage training strategy: cold-start SFT for basic tool selection; self-reflection fine-tuning to strengthen decision revision; and agentic tool reinforcement learning to elicit sophisticated, expert-like diagnostic behaviors. Extensive experiments show that Ophiuchus consistently outperforms both closed-source and open-source SOTA methods across diverse medical benchmarks, including VQA, detection, and reasoning-based segmentation. Our project code is available at https://github.com/SII-zyj/Ophiuchus.

Medical AI models can now explain their answers step by step, but they often still look at medical images too broadly. In difficult cases, the important clue may be a tiny lesion, a subtle boundary, or a small abnormal region that the model needs to inspect more carefully. We introduce Ophiuchus, a medical AI system that can “think with images” by deciding when it needs closer visual evidence, choosing where to look, and using visual tools to inspect that region. Instead of simply calling a tool once, Ophiuchus learns to combine image inspection with its reasoning process, so it can revise its judgment when new visual evidence becomes available. This makes the model better suited for medical tasks that require fine-grained image understanding, such as answering clinical questions, locating important regions, and drawing precise boundaries around abnormalities. Across diverse medical benchmarks, Ophiuchus outperforms strong existing models, suggesting that future medical AI systems may benefit from actively examining images during reasoning rather than relying only on a single global view.