Timezone: »

 
Oral
On the Computation and Communication Complexity of Parallel SGD with Dynamic Batch Sizes for Stochastic Non-Convex Optimization
Hao Yu · rong jin

Wed Jun 12 02:25 PM -- 02:30 PM (PDT) @ Room 104
in Optimization: Convex and Non-convex »
For SGD based distributed stochastic optimization, computation complexity, measured by the convergence rate in terms of the number of stochastic gradient access, and communication complexity, measured by the number of inter-node communication rounds, are the most important two performance metrics. The classical data-parallel implementation of SGD over $N$ workers can achieve a linear speedup of its convergence rate but incurs an inter-node communication round at each batch. We study the benefit of using dynamically increasing batch sizes in parallel SGD for stochastic non-convex optimization by charactering the attained convergence rate and the required number of communication rounds. We show that for stochastic non-convex optimization under the P-L condition, the classical data parallel SGD with exponentially increasing batch sizes can achieve the fastest known $O(1/(NT))$ convergence with linear speedup using only $\log(T)$ communication rounds. For general stochastic non-convex optimization, we propose a Catalyst-like algorithm that achieves the fastest known $O(1/\sqrt{NT})$ convergence with linear speedup using only $O(\sqrt{NT}\log(\frac{T}{N}))$ communication rounds.
[ Slides]  [ Video] 

Author Information

Hao Yu (Alibaba Group (US) Inc)
rong jin (alibaba group)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors