Timezone: »

 
Multimodal Representation Learning of Cardiovascular Magnetic Resonance Imaging
Jielin Qiu · Peide Huang · Makiya Nakashima · Jaehyun Lee · Jiacheng Zhu · Wilson Tang · Pohao Chen · Christopher Nguyen · Byung-Hak Kim · Debbie Kwon · Douglas Weber · Ding Zhao · David Chen

Sat Jul 29 05:00 PM -- 05:10 PM (PDT) @
in Machine Learning for Multimodal Healthcare Data »

Self-supervised learning is crucial for clinical imaging applications, given the lack of explicit labels in healthcare. However, conventional approaches that rely on precise vision-language alignment are not always feasible in complex clinical imaging modalities, such as cardiac magnetic resonance (CMR). CMR provides a comprehensive visualization of cardiac anatomy, physiology, and microstructure, making it challenging to interpret. Additionally, CMR reports require synthesizing information from sequences of images and different views, resulting in potentially weak alignment between the study and diagnosis report pair.To overcome these challenges, we propose \textbf{CMRformer}, a multimodal learning framework to jointly learn sequences of CMR images and associated cardiologist's reports. Moreover, one of the major obstacles to improving CMR study is the lack of large, publicly available datasets. To bridge this gap, we collected a large \textbf{CMR dataset}, which consists of 13,787 studies from clinical cases. By utilizing our proposed CMRformer and our collected dataset, we achieved remarkable performance in real-world clinical tasks, such as CMR image retrieval and diagnosis report retrieval. Furthermore, the learned representations are evaluated to be practically helpful for downstream applications, such as disease classification. Our work could potentially expedite progress in the CMR study and lead to more accurate and effective diagnosis and treatment.

Keywords: Medical Imaging · Multimodal biomarkers · Multimodal fusion
[ Slides] 

Author Information

Jielin Qiu (Carnegie Mellon University)
Peide Huang (Carnegie Mellon University)
Makiya Nakashima (Cleveland Clinic)
Jaehyun Lee (Cleveland Clinic )
Jiacheng Zhu (Carnegie Mellon University)
Wilson Tang (Cleveland Clinic)
Pohao Chen (Cleveland Clinic)
Christopher Nguyen (Cleveland Clinic)
Byung-Hak Kim (AKASA)
Debbie Kwon (Cleveland Clinic)
Douglas Weber (Carnegie Mellon University)
Ding Zhao (Carnegie Mellon University)
David Chen (Cleveland Clinic)

More from the Same Authors