MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models

Medical large vision-language models (Med-LVLMs) have recently achieved remarkable progress in vision–language comprehension and medical image segmentation. However, existing models still struggle to unify these two capabilities, which is essential for achieving clinically reasoning that connects visual findings with semantic interpretation. We present MedSIGHT, a unified framework that equips Med-LVLMs with structured, pixel-level understanding for grounded visual comprehension. MedSIGHT introduces a novel Region Perceiver module that produces region-centric tokens, encoding spatial information directly into representation space of the language model. We further propose a medical region codebook into the LLM vocabulary, allowing the model to generate discrete region codes as symbolic representations of anatomical and pathological regions. These codes are decoded through the Region Perceiver to reconstruct segmentation mask, achieving end-to-end spatial grounding. Lastly, MedSIGHT combines Region Perceiver, Codebook and LLM using our proposed progressive training strategy to gradually aligns these modules stably. Trained on only 72K multimodal instruction pairs, MedSIGHT achieves state-of-the-art performance across diverse imaging modalities on both medical comprehension and segmentation tasks. Code and model are publicly available at GitHub.

Medical AI systems can answer questions about medical images and identify regions such as organs or lesions, but most existing models struggle to combine these abilities into unified clinical reasoning. As a result, they may generate answers without clearly grounding them in the image. We present MedSIGHT, a new framework that enables medical vision-language models to understand medical images at both the semantic and pixel level. MedSIGHT learns structured representations of important anatomical and pathological regions, allowing the model not only to answer medical questions but also to localize the visual evidence supporting its reasoning. To achieve this, we introduce a new region-based representation mechanism and a symbolic coding system that connects image regions with medical concepts. We also design a progressive training strategy that helps the model learn visual grounding and reasoning together effectively. Experiments across diverse medical imaging tasks show that MedSIGHT achieves state-of-the-art performance while providing more reliable and interpretable visual understanding.