Predicting evolutionary rate as a pretraining task improves genome language model representations

Genome language models (gLM) have the potential to further understanding of regulatory genomics without requiring labeled data. Most gLMs are pretrained using sequence reconstruction tasks inspired by natural language processing, but recent studies have shown that these gLMs often fail to capture biological signal. To overcome this, we introduce pretraining tasks that predict the rate of evolution. These tasks are designed so that they can be composed with sequence reconstruction, enabling a controlled comparison of predicting sequence only, evolutionary rate only, or both. To address gaps in existing evaluations, we developed a suite of biologically grounded benchmarks. Across these tasks, and for established variant effect prediction benchmarks, models pretrained on both sequence and evolutionary rate outperform those trained on sequence alone, and training on evolutionary rate can make the even the relatively small models in our work competitive with much larger existing gLMs for some tasks on the human genome. These results establish evolution as a key training target for genome-scale models.

Genome language models are deep learning models trained on DNA, with the goal of learning useful representations of the genome in the same way that language models learn useful representations of text. Most of these models are trained by reconstructing DNA sequence, similar to how language models learn by predicting missing or next words. However, recent studies suggest that this training strategy does not always recover known biological signals. We introduce a new way to train genome language models, instead of only reconstructing DNA, the models also predict how evolutionarily conserved or accelerated each DNA base pair in the human genome is. This is useful because evolution acts as a proxy for function: important genomic positions often change more slowly across species. Our method lets us directly compare models trained on sequence alone, evolution alone, or both sequence and evolution. We also develop biologically grounded evaluations to address gaps in existing benchmarks for genome language models. Across these evaluations, and on established variant effect prediction tasks, models trained with evolution outperform models trained on sequence alone. In some cases, even our relatively small models become competitive with much larger genome language models on the human genome. These results suggest that evolution should be a central training signal for genome-scale models.