Timezone: »

 
Poster
Compressing Gradient Optimizers via Count-Sketches
Ryan Spring · Anastasios Kyrillidis · Vijai Mohan · Anshumali Shrivastava

Tue Jun 11 06:30 PM -- 09:00 PM (PDT) @ Pacific Ballroom #83
in Posters Tue »

Many popular first-order optimization methods accelerate the convergence rate of deep learning models. However, these algorithms require auxiliary variables, which cost additional memory proportional to the number of parameters in the model. The problem is becoming more severe as models grow larger to learn from complex, large-scale datasets. Our proposed solution is to maintain a linear sketch to compress the auxiliary variables. Our approach has the same performance as the full-sized baseline, while using less space for the auxiliary variables. Theoretically, we prove that count-sketch optimization maintains the SGD convergence rate, while gracefully reducing memory usage for large-models. We show a rigorous evaluation on popular architectures such as ResNet-18 and Transformer-XL. On the 1-Billion Word dataset, we save 25% of the memory used during training (7.7 GB instead of 10.8 GB) with minimal accuracy and performance loss. For an Amazon extreme classification task with over 49.5 million classes, we also reduce the training time by 38%, by increasing the mini-batch size 3.5x using our count-sketch optimizer.

Keywords: Large Scale Learning and Big Data · Natural Language Processing · Non-convex Optimization · Parallel and Distributed Learning · Recommender Systems

Author Information

Ryan Spring (Rice University)
Anastasios Kyrillidis (Rice University)
Vijai Mohan (www.amazon.com)
Anshumali Shrivastava (Rice University)

Anshumali Shrivastava is an associate professor in the computer science department at Rice University. His broad research interests include randomized algorithms for large-scale machine learning. In 2018, Science news named him one of the Top-10 scientists under 40 to watch. He is a recipient of National Science Foundation CAREER Award, a Young Investigator Award from Air Force Office of Scientific Research, and machine learning research award from Amazon. His research on hashing inner products has won Best Paper Award at NIPS 2014 while his work on representing graphs got the Best Paper Award at IEEE/ACM ASONAM 2014. Anshumali finished his Ph.D. in 2015 from Cornell University.

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors