Timezone: »

 
Poster
Delay-Adaptive Step-sizes for Asynchronous Learning
Xuyang Wu · Sindri Magnússon · Hamid Reza Feyzmahdavian · Mikael Johansson

Thu Jul 21 03:00 PM -- 05:00 PM (PDT) @ Hall E #703
in Poster Session 3 »

In scalable machine learning systems, model training is often parallelized over multiple nodes that run without tight synchronization. Most analysis results for the related asynchronous algorithms use an upper bound on the information delays in the system to determine learning rates. Not only are such bounds hard to obtain in advance, but they also result in unnecessarily slow convergence. In this paper, we show that it is possible to use learning rates that depend on the actual time-varying delays in the system. We develop general convergence results for delay-adaptive asynchronous iterations and specialize these to proximal incremental gradient descent and block coordinate descent algorithms. For each of these methods, we demonstrate how delays can be measured on-line, present delay-adaptive step-size policies, and illustrate their theoretical and practical advantages over the state-of-the-art.

Keywords: OPT: Convex · OPT: First-order · OPT: Learning for Optimization · OPT: Optimization and Learning under Uncertainty · OPT: Large Scale, Parallel and Distributed · T: Optimization
[ Slides]  [ Poster]  [ Paper PDF] 

Author Information

Xuyang Wu (KTH Royal Institute of Technology)
Sindri Magnússon (Stockholm University)
Hamid Reza Feyzmahdavian (ABB)
Mikael Johansson (KTH Royal Institute of Technology)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors