Bimodal masked language modeling for bulk RNA-seq and DNA methylation representation learning

Oncologists are increasingly relying on multiple modalities to model the complexity of diseases. Within this landscape, transcriptomic and epigenetic data have proven to be particularly instrumental and play an increasingly vital role in clinical applications. However, their integration into multimodal models remains a challenge, especially considering their high dimensionality. In this work, we present a novel bimodal model that jointly learns representations of bulk RNA-seq and DNA methylation leveraging self-supervision from masked language modeling. We implement an architecture that reduces the memory footprint usually attributed to purely transformer-based models when dealing with long sequences. We demonstrate that the obtained bimodal embeddings can be used to fine-tune cancer-type classification and survival models that achieve state-of-the-art performance compared to unimodal models. Furthermore, we introduce a robust learning framework that maintains downstream task performance despite missing modalities, enhancing the model’s applicability in real-world clinical settings.

When doctors treat cancer, they increasingly rely on multiple types of biological information, such as which genes are active and how DNA is modified. While these details are crucial for understanding complex diseases, combining such massive, high-dimensional datasets into a single predictive tool remains a major challenge. To solve this, we developed a new artificial intelligence system that learns to "read" two types of biological data simultaneously. Similar to how language models learn to complete sentences by filling in missing words, our system uncovers patterns by predicting hidden biological features. We also designed a unique architecture that compresses this enormous amount of data, preventing the system from running out of memory. As a result, our model significantly outperforms existing single-data tools at classifying cancer types and predicting patient survival times. Most importantly, our framework remains highly accurate even if a patient is missing some of this biological data. By handling incomplete real-world data gracefully, our approach brings advanced, multi-layered diagnostic models one step closer to everyday clinical use.