Timezone: »

 
Oral
Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization
zhenxun zhuang · Ashok Cutkosky · Francesco Orabona

Wed Jun 12 02:35 PM -- 02:40 PM (PDT) @ Room 104
in Optimization: Convex and Non-convex »

Stochastic Gradient Descent (SGD) has played a central role in machine learning. However, it requires a carefully hand-picked stepsize for fast convergence, which is notoriously tedious and time-consuming to tune. Over the last several years, a plethora of adaptive gradient-based algorithms have emerged to ameliorate this problem. In this paper, we propose new surrogate losses to cast the problem of learning the optimal stepsizes for the stochastic optimization of a non-convex smooth objective function onto an online convex optimization problem. This allows the use of no-regret online algorithms to compute optimal stepsizes on the fly. In turn, this results in a SGD algorithm with self-tuned stepsizes that guarantees convergence rates that are automatically adaptive to the level of noise.

[ Slides]  [ Video] 

Author Information

zhenxun zhuang (Boston University)
Ashok Cutkosky (Google)
Francesco Orabona (Stony Brook University)
Francesco Orabona

Francesco Orabona is an Associate Professor at KAUST. His background covers both theoretical and practical aspects of machine learning and optimization. His current research interests lie in online learning, and more generally the problem of designing and analyzing adaptive and parameter-free learning algorithms. He received the PhD degree in Electrical Engineering at the University of Genoa in 2007. He is (co)author of more than 60 peer reviewed papers.

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors