Timezone: »
We present GNNAutoScale (GAS), a framework for scaling arbitrary message-passing GNNs to large graphs. GAS prunes entire sub-trees of the computation graph by utilizing historical embeddings from prior training iterations, leading to constant GPU memory consumption in respect to input node size without dropping any data. While existing solutions weaken the expressive power of message passing due to sub-sampling of edges or non-trainable propagations, our approach is provably able to maintain the expressive power of the original GNN. We achieve this by providing approximation error bounds of historical embeddings and show how to tighten them in practice. Empirically, we show that the practical realization of our framework, PyGAS, an easy-to-use extension for PyTorch Geometric, is both fast and memory-efficient, learns expressive node representations, closely resembles the performance of their non-scaling counterparts, and reaches state-of-the-art performance on large-scale graphs.
Author Information
Matthias Fey (TU Dortmund University)
Jan Eric Lenssen (TU Dortmund)
Frank Weichert (Technical University of Dortmund)
Jure Leskovec (Stanford University)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Spotlight: GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddings »
Thu. Jul 22nd 12:35 -- 12:40 PM Room
More from the Same Authors
-
2020 : (#45 / Sess. 1) Hierarchical Inter-Message Passing for Learning on Molecular Graphs »
Matthias Fey -
2022 : LinkBERT: Language Model Pretraining with Document Link Knowledge »
Michihiro Yasunaga · Jure Leskovec · Percy Liang -
2023 : Retrieval-Augmented Multimodal Language Modeling »
Michihiro Yasunaga · Armen Aghajanyan · Weijia Shi · Rich James · Jure Leskovec · Percy Liang · Mike Lewis · Luke Zettlemoyer · Wen-tau Yih -
2023 : PRODIGY: Enabling In-context Learning Over Graphs »
Qian Huang · Hongyu Ren · Peng Chen · Gregor Kržmanc · Daniel Zeng · Percy Liang · Jure Leskovec -
2023 : Learning Large Graph Property Prediction via Graph Segment Training »
Kaidi Cao · Phitchaya Phothilimthana · Sami Abu-El-Haija · Dustin Zelle · Yanqi Zhou · Charith Mendis · Jure Leskovec · Bryan Perozzi -
2023 Poster: Geometric Latent Diffusion Models for 3D Molecule Generation »
Minkai Xu · Alexander Powers · Ron Dror · Stefano Ermon · Jure Leskovec -
2023 Poster: Retrieval-Augmented Multimodal Language Modeling »
Michihiro Yasunaga · Armen Aghajanyan · Weijia Shi · Richard James · Jure Leskovec · Percy Liang · Mike Lewis · Luke Zettlemoyer · Scott Yih -
2021 Poster: WILDS: A Benchmark of in-the-Wild Distribution Shifts »
Pang Wei Koh · Shiori Sagawa · Henrik Marklund · Sang Michael Xie · Marvin Zhang · Akshay Balsubramani · Weihua Hu · Michihiro Yasunaga · Richard Lanas Phillips · Irena Gao · Tony Lee · Etienne David · Ian Stavness · Wei Guo · Berton Earnshaw · Imran Haque · Sara Beery · Jure Leskovec · Anshul Kundaje · Emma Pierson · Sergey Levine · Chelsea Finn · Percy Liang -
2021 Oral: WILDS: A Benchmark of in-the-Wild Distribution Shifts »
Pang Wei Koh · Shiori Sagawa · Henrik Marklund · Sang Michael Xie · Marvin Zhang · Akshay Balsubramani · Weihua Hu · Michihiro Yasunaga · Richard Lanas Phillips · Irena Gao · Tony Lee · Etienne David · Ian Stavness · Wei Guo · Berton Earnshaw · Imran Haque · Sara Beery · Jure Leskovec · Anshul Kundaje · Emma Pierson · Sergey Levine · Chelsea Finn · Percy Liang -
2021 Poster: LEGO: Latent Execution-Guided Reasoning for Multi-Hop Question Answering on Knowledge Graphs »
Hongyu Ren · Hanjun Dai · Bo Dai · Xinyun Chen · Michihiro Yasunaga · Haitian Sun · Dale Schuurmans · Jure Leskovec · Denny Zhou -
2021 Spotlight: LEGO: Latent Execution-Guided Reasoning for Multi-Hop Question Answering on Knowledge Graphs »
Hongyu Ren · Hanjun Dai · Bo Dai · Xinyun Chen · Michihiro Yasunaga · Haitian Sun · Dale Schuurmans · Jure Leskovec · Denny Zhou -
2020 : Graph Neural Networks for Reasoning over Multimodal Content »
Jure Leskovec -
2020 Workshop: Bridge Between Perception and Reasoning: Graph Neural Networks & Beyond »
Jian Tang · Le Song · Jure Leskovec · Renjie Liao · Yujia Li · Sanja Fidler · Richard Zemel · Ruslan Salakhutdinov -
2020 : Invited Talk 7 Q&A - Jure Leskovec »
Jure Leskovec -
2020 : Invited Talk 7 - Generalizing to Novel Tasks in the Low-Data Regime - Jure Leskovec »
Jure Leskovec -
2020 : Update: Open Graph Benchmark »
Jure Leskovec -
2020 : Update: PyTorch Geometric »
Matthias Fey -
2020 Workshop: Graph Representation Learning and Beyond (GRL+) »
Petar Veličković · Michael M. Bronstein · Andreea Deac · Will Hamilton · Jessica Hamrick · Milad Hashemi · Stefanie Jegelka · Jure Leskovec · Renjie Liao · Federico Monti · Yizhou Sun · Kevin Swersky · Rex (Zhitao) Ying · Marinka Zitnik -
2020 Poster: Coresets for Data-efficient Training of Machine Learning Models »
Baharan Mirzasoleiman · Jeff Bilmes · Jure Leskovec -
2020 Poster: Graph Structure of Neural Networks »
Jiaxuan You · Jure Leskovec · Kaiming He · Saining Xie -
2020 Poster: Learning to Simulate Complex Physics with Graph Networks »
Alvaro Sanchez-Gonzalez · Jonathan Godwin · Tobias Pfaff · Rex (Zhitao) Ying · Jure Leskovec · Peter Battaglia -
2019 Poster: Position-aware Graph Neural Networks »
Jiaxuan You · Rex (Zhitao) Ying · Jure Leskovec -
2019 Oral: Position-aware Graph Neural Networks »
Jiaxuan You · Rex (Zhitao) Ying · Jure Leskovec -
2018 Poster: GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models »
Jiaxuan You · Rex (Zhitao) Ying · Xiang Ren · Will Hamilton · Jure Leskovec -
2018 Oral: GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models »
Jiaxuan You · Rex (Zhitao) Ying · Xiang Ren · Will Hamilton · Jure Leskovec