Timezone: »
Stochastic rounding is a critical technique used in low-precision deep neural networks (DNNs) training to ensure good model accuracy. However, it requires a large number of random numbers generated on the fly. This is not a trivial task on the hardware platforms such as FPGA and ASIC.The widely used solution is to introduce random number generators with extra hardware costs. In this paper, we innovatively propose to employ the stochastic property of DNN training process itself and directly extract random numbers from DNNs in a self-sufficient manner. We propose different methods to obtain random numbers from different sources in neural networks and a generator-free framework is proposed for low-precision DNN training on a variety of deep learning tasks. Moreover, we evaluate the quality of the extracted random numbers and find that high-quality random numbers widely exist in DNNs, while their quality can even pass the NIST test suite.
Author Information
Geng Yuan (Northeastern University)
Sung-En Chang (Northeastern University)
Alec Lu (Simon Fraser University )
Jun Liu (Carnegie Mellon University)
Yanyu Li (Northeastern University)
Yushu Wu (Northeastern University)
Zhenglun Kong (Northeastern University)
Yanyue Xie (Northeastern University)
Peiyan Dong (Northeastern University)
Minghai Qin (Western Digital Research)
Xiaolong Ma (Northeastern University)
Zhenman Fang (Simon Fraser University)
Yanzhi Wang (Northeastern University)
More from the Same Authors
-
2023 Poster: DualHSIC: HSIC-Bottleneck and Alignment for Continual Learning »
Zifeng Wang · Zheng Zhan · Yifan Gong · Yucai Shao · Stratis Ioannidis · Yanzhi Wang · Jennifer Dy -
2023 Poster: SpeedDETR: Speed-aware Transformers for End-to-end Object Detection »
Peiyan Dong · Zhenglun Kong · Xin Meng · PENG ZHANG · hao tang · Yanzhi Wang · Chih-Hsien Chou -
2022 : Low-Bit DNN Training with Hardware-Efficient Stochastic Rounding Unit Design »
Sung-En Chang · Geng Yuan · Alec Lu · Mengshu Sun · Yanyu Li · Xiaolong Ma · Yanyue Xie · Minghai Qin · Xue Lin · Zhenman Fang · Yanzhi Wang -
2022 Poster: Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets »
Tianlong Chen · Xuxi Chen · Xiaolong Ma · Yanzhi Wang · Zhangyang “Atlas” Wang -
2022 Spotlight: Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets »
Tianlong Chen · Xuxi Chen · Xiaolong Ma · Yanzhi Wang · Zhangyang “Atlas” Wang -
2021 Poster: Lottery Ticket Preserves Weight Correlation: Is It Desirable or Not? »
Ning Liu · Geng Yuan · Zhengping Che · Xuan Shen · Xiaolong Ma · Qing Jin · Jian Ren · Jian Tang · Sijia Liu · Yanzhi Wang -
2021 Spotlight: Lottery Ticket Preserves Weight Correlation: Is It Desirable or Not? »
Ning Liu · Geng Yuan · Zhengping Che · Xuan Shen · Xiaolong Ma · Qing Jin · Jian Ren · Jian Tang · Sijia Liu · Yanzhi Wang