SWALP : Stochastic Weight Averaging in Low Precision Training
Guandao Yang · Tianyi Zhang · Polina Kirichenko · Junwen Bai · Andrew Wilson · Christopher De Sa

Tue Jun 11th 06:30 -- 09:00 PM @ Pacific Ballroom #58

Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precision SGD even with all numbers quantized down to 8 bits, including the gradient accumulators. Additionally, we show that SWALP converges arbitrarily close to the optimal solution for quadratic objectives, and to a noise ball asymptotically smaller than low precision SGD in strongly convex settings.

Author Information

Guandao Yang (Cornell University)
Tianyi Zhang (Cornell University)
Polina Kirichenko (Cornell)
Junwen Bai (Cornell)
Andrew Wilson (Cornell University)
Christopher De Sa (Cornell)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors