Timezone: »

Exploring Hidden Dimensions in Accelerating Convolutional Neural Networks
Zhihao Jia · Sina Lin · Charles Qi · Alex Aiken

Thu Jul 12 09:15 AM -- 12:00 PM (PDT) @ Hall B #34

The past few years have witnessed growth in the computational requirements for training deep convolutional neural networks. Current approaches parallelize training onto multiple devices by applying a single parallelization strategy (e.g., data or model parallelism) to all layers in a network. Although easy to reason about, these approaches result in suboptimal runtime performance in large-scale distributed training, since different layers in a network may prefer different parallelization strategies. In this paper, we propose layer-wise parallelism that allows each layer in a network to use an individual parallelization strategy. We jointly optimize how each layer is parallelized by solving a graph search problem. Our evaluation shows that layer-wise parallelism outperforms state-of-the-art approaches by increasing training throughput, reducing communication costs, achieving better scalability to multiple GPUs, while maintaining original network accuracy.

Author Information

Zhihao Jia (Stanford University)
Sina Lin (Microsoft)
Charles Qi (Stanford University)
Alex Aiken (Stanford University)

Related Events (a corresponding poster, oral, or spotlight)