Timezone: »

 
Learning Large Graph Property Prediction via Graph Segment Training
Kaidi Cao · Phitchaya Phothilimthana · Sami Abu-El-Haija · Dustin Zelle · Yanqi Zhou · Charith Mendis · Jure Leskovec · Bryan Perozzi

Fri Jul 28 02:10 PM -- 03:00 PM (PDT) @
in 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML) »
Event URL: https://openreview.net/forum?id=Emy5KJ4w6J »

Learning to predict properties of large graphs is challenging because each prediction requires the knowledge of an entire graph, while the amount of memory available during training is bounded. Here we propose Graph Segment Training (GST), a general framework that utilizes a divide-and-conquer approach to allow learning large graph property prediction with a constant memory footprint. GST first divides a large graph into segments and then backpropagates through only a few segments sampled per training iteration. We refine the GST paradigm by introducing a historical embedding table to efficiently obtain embeddings for segments not sampled for backpropagation. To mitigate the staleness of historical embeddings, we design two novel techniques. First, we finetune the prediction head to fix the input distribution shift. Second, we introduce Stale Embedding Dropout to drop some stale embeddings during training to reduce bias. We evaluate our complete method GST-EFD (with all the techniques together) on two large graph property prediction benchmarks: MalNet and TpuGraphs. Our experiments show that GST-EFD is both memory-efficient and fast, while offering a slight boost on test accuracy over a typical full graph training regime.

Author Information

Kaidi Cao (Stanford University)
Phitchaya Phothilimthana (Google)
Sami Abu-El-Haija (USC Information Sciences Institute)
Dustin Zelle (Google LLC)
Yanqi Zhou (Google)
Charith Mendis (University of Illinois at Urbana-Champaign)
Jure Leskovec (Stanford University)
Bryan Perozzi (Google Research)

More from the Same Authors