Timezone: »
Activity and property prediction models are the central workhorses in drug discovery and materials sciences, but currently, they have to be trained or fine-tuned for new tasks. Without training or fine-tuning, scientific language models could be used for such low-data tasks through their announced zero- and few-shot capabilities. However, their predictive quality at activity prediction is lacking. In this work, we envision a novel type of activity prediction model that is able to adapt to new prediction tasks at inference time, via understanding textual information describing the task. To this end, we propose a new architecture with separate modules for chemical and natural language inputs, and a contrastive pretraining objective on data from large biochemical databases. In extensive experiments, we show that our method CLAMP yields improved predictive performance on few-shot learning benchmarks and zero-shot problems in drug discovery. We attribute the advances of our method to the modularized architecture and to our pre-training objective.
Author Information
Philipp Seidl (Johannes Kepler University Linz)
Andreu Vall (Johannes Kepler University Linz)
Sepp Hochreiter (ELLIS Unit Linz, LIT AI Lab, Institute for Machine Learning, Johannes Kepler University, Institute for Advanced Research in Artificial Intelligence (IARAI))
Sepp Hochreiter is heading the Institute for Machine Learning, the ELLIS Unit Linz, the LIT AI Lab at the JKU Linz and is director of private research institute IARAI. He is a pioneer of Deep Learning as he discovered the famous problem of vanishing or exploding gradients and invented the long short-term memory (LSTM).
Günter Klambauer (Johannes Kepler University Linz Austria)
More from the Same Authors
-
2022 Poster: History Compression via Language Models in Reinforcement Learning »
Fabian Paischer · Thomas Adler · Vihang Patil · Angela Bitto-Nemling · Markus Holzleitner · Sebastian Lehner · Hamid Eghbal-zadeh · Sepp Hochreiter -
2022 Spotlight: History Compression via Language Models in Reinforcement Learning »
Fabian Paischer · Thomas Adler · Vihang Patil · Angela Bitto-Nemling · Markus Holzleitner · Sebastian Lehner · Hamid Eghbal-zadeh · Sepp Hochreiter -
2022 Poster: Align-RUDDER: Learning From Few Demonstrations by Reward Redistribution »
Vihang Patil · Markus Hofmarcher · Marius-Constantin Dinu · Matthias Dorfer · Patrick Blies · Johannes Brandstetter · Jose A. Arjona-Medina · Sepp Hochreiter -
2022 Oral: Align-RUDDER: Learning From Few Demonstrations by Reward Redistribution »
Vihang Patil · Markus Hofmarcher · Marius-Constantin Dinu · Matthias Dorfer · Patrick Blies · Johannes Brandstetter · Jose A. Arjona-Medina · Sepp Hochreiter -
2021 Spotlight: MC-LSTM: Mass-Conserving LSTM »
Pieter-Jan Hoedt · Frederik Kratzert · Daniel Klotz · Christina Halmich · Markus Holzleitner · Grey Nearing · Sepp Hochreiter · Günter Klambauer -
2021 Poster: MC-LSTM: Mass-Conserving LSTM »
Pieter-Jan Hoedt · Frederik Kratzert · Daniel Klotz · Christina Halmich · Markus Holzleitner · Grey Nearing · Sepp Hochreiter · Günter Klambauer