LLM-Guided Diagnostic Evidence Alignment for Medical Vision–Language Pretraining under Limited Pairing

Most existing CLIP-style medical vision--language pretraining methods rely on global or local alignment with substantial paired data. However, global alignment is easily dominated by non-diagnostic information, while local alignment fails to integrate key diagnostic evidence. As a result, learning reliable diagnostic representations becomes difficult, which limits their applicability in medical scenarios with limited paired data. To address this issue, we propose an LLM-Guided Diagnostic Evidence Alignment method (LGDEA), which shifts the pretraining objective toward evidence-level alignment that is more consistent with the medical diagnostic process. Specifically, we leverage LLMs to extract key diagnostic evidence from radiology reports and construct a shared diagnostic evidence space, enabling evidence-aware cross-modal alignment and allowing LGDEA to effectively exploit abundant unpaired medical images and reports, thereby substantially alleviating the reliance on paired data. Extensive experimental results demonstrate that our method achieves consistent and significant improvements on phrase grounding, image--text retrieval, and zero-shot classification, and even rivals pretraining methods that rely on substantial paired data.

Medical AI systems are often trained by matching chest X-ray images with their radiology reports. Existing methods usually compare the whole image with the whole report, or match small image regions with individual words. However, the first strategy can be distracted by information that is not useful for diagnosis, while the second may focus on scattered details and miss the key evidence doctors actually use. We propose LGDEA, a method that helps medical AI models learn from diagnostic evidence rather than from coarse image-report matching alone. LGDEA uses large language models to identify important diagnostic evidence from radiology reports and organizes this evidence into a shared space that connects images and text. This allows the model to focus on clinically meaningful findings, such as disease signs and relevant image regions. Importantly, LGDEA can also make better use of many unpaired medical images and reports, reducing the need for large amounts of manually paired data. Experiments show that LGDEA improves performance on medical grounding, image-text retrieval, and zero-shot classification, and can even compete with methods trained using much more paired data.