MESH-HR: Multimodal Fusion of Somatic DNA Profiles and Histopathology for Continuous Breast Cancer Receptor Subtyping via LLM-Assisted Annotation

Breast cancer receptor subtyping determines eligibility for targeted therapies based on estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2) expression, which are typically assessed via immunohistochemistry (IHC). We introduce MESH-HR (Multimodal Ensemble of Somatic Variants and H\&E Slides for Hormone Receptor subtyping), a multimodal framework that integrates two biological modalities, somatic DNA profiles and whole-slide histopathology images, to predict receptor status probabilities without IHC. To enable large-cohort training without manual curation, ground truth receptor labels are extracted at scale from unstructured pathology reports using a GPT-4–based LLM pipeline. MESH-HR combines an attention-based multiple instance learning (ABMIL) vision encoder with an XGBoost model over somatic features, fusing complementary signal across DNA and imaging modalities. Trained on $>$1,300 breast cancers, MESH-HR achieves AUCs of 0.90 (ER), 0.84 (PR), and 0.96 (HER2) on held-out data, outperforming unimodal baselines, and generalizes zero-shot to The Cancer Genome Atlas Breast Cancer cohort (TCGA-BRCA). Continuous probabilistic outputs improve survival stratification over binary clinical labels, recovering prognostic signal lost under discrete IHC thresholds. We further apply MESH-HR zero-shot to cancers of unknown primary (CUP), a population in which receptor profiling is rarely performed due to the absence of a known tissue of origin, obtaining biologically consistent, survival-predictive receptor estimates that support receptor-informed personalized treatment stratification.