Do Multiple Instance Learning Models Transfer?

Multiple Instance Learning (MIL) is a cornerstone approach in computational pathology for distilling embeddings from gigapixel tissue images into patient-level representations to predict clinical outcomes. However, MIL is frequently challenged by the constraints of working with small, weakly-supervised clinical datasets. Unlike fields such as natural language processing and computer vision, which effectively use transfer learning to improve model quality in data-scarce environments, the transferability of MIL models remains largely unexplored. We conduct the first comprehensive investigation into transfer learning capabilities of pretrained MIL models, evaluating 11 MIL models across 19 pretraining tasks spanning tissue subtyping, cancer grading, and molecular subtype prediction. We observe a substantial performance boost with finetuning pretrained models over training from randomly initialized weights, even with domain differences between pretraining and target tasks. Pretraining on pan-cancer datasets enables consistent generalization across organs and task types compared to single-disease pretraining. Remarkably, this pan-cancer pretraining leads to better transfer than that of a state-of-the-art slide-level foundation model, while using only 6.5\% of the training data. These findings indicate that MIL architectures exhibit robust adaptability, offering insights into the benefits of leveraging pretrained models to enhance performance in computational pathology.

Analyzing gigapixel images of human tissue is crucial for disease diagnosis. Multiple Instance learning (MIL) provides a means of consolidating the key insights from these massive images into a condensed embedding, but often exhibits poor generalizability due to insufficient training data. While transfer learning is a common remedy for data scarcity in other ML fields, MIL models are typically trained from scratch in our field, despite the challenges of data scarcity. Here, we investigate the feasability of whether transfer learning of MIL models can be leveraged to improve model generalization. We thoroughly investigate whether MIL models trained on one task can be transferred to new tasks, training 11 different MIL models on 21 total pretraining tasks spanning breast, brain, lung, and prostate cancer, as well as a challenging 108-class classification task across 19 cancer types.This allowed us to gain insights into how different attributes of the MIL model, pretrain task, and target task, affect transfer performance. Our research shows that pretrained MIL models perform much better than models trained from scratch, even if their initial training was on a different organ or disease. Importantly, models trained on the challenging pancancer data were able to generalize best across different organs and tasks, while using substantially less data than other self-supervised learning-based foundation models. These findings highlight MIL models as highly adaptable, and supervised pretraining as an effective and accessible means for addressing the challenges of learning clinically-meaningful representations from challenging tasks and data scarce regimes. We also provide a GitHub library to standardize MIL model initialization and loading model weights, available at https://github.com/mahmoodlab/MIL-Lab