Timezone: »

 
Poster
Don't Waste Your Bits! Squeeze Activations and Gradients for Deep Neural Networks via TinyScript
Fangcheng Fu · Yuzheng Hu · Yihan He · Jiawei Jiang · Yingxia Shao · Ce Zhang · Bin Cui

Wed Jul 15 05:00 AM -- 05:45 AM & Wed Jul 15 04:00 PM -- 04:45 PM (PDT) @
in Poster Session 21 »

Recent years have witnessed intensive research interests on training deep neural networks (DNNs) more efficiently by quantization-based compression methods, which facilitate DNNs training in two ways: (1) activations are quantized to shrink the memory consumption, and (2) gradients are quantized to decrease the communication cost. However, existing methods mostly use a uniform mechanism that quantizes the values evenly. Such a scheme may cause a large quantization variance and slow down the convergence in practice.

In this work, we introduce TinyScript, which applies a non-uniform quantization algorithm to both activations and gradients. TinyScript models the original values by a family of Weibull distributions and searches for ''quantization knobs'' that minimize quantization variance. We also discuss the convergence of the non-uniform quantization algorithm on DNNs with varying depths, shedding light on the number of bits required for convergence. Experiments show that TinyScript always obtains lower quantization variance, and achieves comparable model qualities against full precision training using 1-2 bits less than the uniform-based counterpart.

Keywords: Large Scale Learning and Big Data · Non-convex Optimization · Optimization - Large Scale, Parallel and Distributed

Author Information

Fangcheng Fu (Peking University)
Yuzheng Hu (Peking University)
Yihan He (Peking University)

Undergraduate at Peking University 2020 class working on Machine Learning and Data Privacy

Jiawei Jiang (ETH Zurich)
Yingxia Shao (BUPT)
Ce Zhang (ETH Zurich)
Bin Cui (Peking University)

More from the Same Authors