Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography

Vision–language pre-training (VLP) holds great promise for general-purpose medical AI by leveraging radiology reports as rich textual supervision, yet existing methods struggle with 3D CT imaging due to inefficient visual backbones and coarse semantic alignment. To address these issues, we propose a tailored VLP framework featuring three key components: (1) a CNN–ViT hybrid encoder that replaces ViT’s patch embedding with a 3D CNN backbone to efficiently capture local anatomical details while preserving global attention and compatibility with pre-trained cross-modal priors; (2) a disease-level contrastive learning mechanism using learnable query tokens to dynamically extract disease-specific semantics from full reports and align them with corresponding visual features, thereby disentangling distinct diseases within the same anatomical region; and (3) a diagnosis-aware prompt strategy that employs real clinical phrases and aggregated disease prototypes to bridge the pre-training–inference gap and enhance zero-shot diagnostic reliability. Our model achieves state-of-the-art performance on CT-RATE (84.4\% AUC, +5.1%) and Rad-ChestCT (75.4\% AUC, +5.4%), with even larger gains (+9.8% AUC) on a challenging 60-disease benchmark, and demonstrates strong transferability to radiology report generation, underscoring the generality and clinical utility of our approach.

Modern AI systems can help doctors diagnose diseases from medical scans, but they struggle with 3D CT images because these are complex and computationally demanding. Existing methods also fail to distinguish different diseases affecting the same organ, such as telling apart pneumonia from lung nodules. To solve this, we developed a new AI framework that combines two powerful techniques: a hybrid visual encoder that efficiently processes 3D anatomy, and a disease-focused learning mechanism that links specific medical conditions to their visual patterns in scans. Our approach also uses real clinical language from radiology reports instead of simplified templates, making diagnoses more reliable. Tested on thousands of chest CT scans, our model outperforms previous methods by over 5% in diagnostic accuracy, and shows even larger gains when identifying 60 different conditions. This work brings us closer to AI tools that understand medical images with the nuance of real clinical reasoning, potentially improving diagnostic accuracy and accessibility.