Timezone: »

 
Protein Representation Learning by Geometric Structure Pretraining
Zuobai Zhang · Zuobai Zhang · Minghao Xu · Minghao Xu · Arian Jamasb · Arian Jamasb · Vijil Chenthamarakshan · Vijil Chenthamarakshan · Aurelie Lozano · Payel Das · Payel Das · Jian Tang · Jian Tang

@
in The First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward »
Event URL: https://openreview.net/forum?id=V5MEFikiBQy »

Learning effective protein representations is critical in a variety of tasks in biology such as predicting protein function or structure. Existing approaches usually pretrain protein language models on a large number of unlabeled amino acid sequences and then finetune the models with some labeled data in downstream tasks. Despite the effectiveness of sequence-based approaches, the power of pretraining on known protein structures, which are available in smaller numbers only, has not been explored for protein property prediction, though protein structures are known to be determinants of protein function. In this paper, we propose to pretrain protein representations according to their 3D structures. We first present a simple yet effective encoder to learn the geometric features of a protein. We pretrain the protein graph encoder by leveraging multiview contrastive learning and different self-prediction tasks. Experimental results on both function prediction and fold classification tasks show that our proposed pretraining methods outperform or are on par with the state-of-the-art sequence-based methods, while using much less data. All codes and models will be published upon acceptance.

Author Information

Zuobai Zhang (Mila)
Zuobai Zhang (Mila)
Minghao Xu (Montreal Institute for Learning Algorithms, University of Montreal, University of Montreal)
Minghao Xu (Montreal Institute for Learning Algorithms, University of Montreal, University of Montreal)
Arian Jamasb (University of Cambridge)
Arian Jamasb (University of Cambridge)
Vijil Chenthamarakshan (IBM Research)
Vijil Chenthamarakshan (IBM Research)
Aurelie Lozano (IBM)
Payel Das (IBM Research AI)
Payel Das (IBM Research AI)
Jian Tang (Mila)
Jian Tang (Mila)

More from the Same Authors