Skip to yearly menu bar Skip to main content


Learning-to-Learn Stochastic Gradient Descent with Biased Regularization

Giulia Denevi · Carlo Ciliberto · Riccardo Grazzi · Massimiliano Pontil

Pacific Ballroom #257

Keywords: [ Transfer and Multitask Learning ] [ Supervised Learning ] [ Statistical Learning Theory ] [ Optimization - Others ] [ Online Learning ]


We study the problem of learning-to-learn: infer- ring a learning algorithm that works well on a family of tasks sampled from an unknown distribution. As class of algorithms we consider Stochastic Gradient Descent (SGD) on the true risk regularized by the square euclidean distance from a bias vector. We present an average excess risk bound for such a learning algorithm that quantifies the potential benefit of using a bias vector with respect to the unbiased case. We then propose a novel meta-algorithm to estimate the bias term online from a sequence of observed tasks. The small memory footprint and low time complexity of our approach makes it appealing in practice while our theoretical analysis provides guarantees on the generalization properties of the meta-algorithm on new tasks. A key feature of our results is that, when the number of tasks grows and their vari- ance is relatively small, our learning-to-learn approach has a significant advantage over learning each task in isolation by standard SGD without a bias term. Numerical experiments demonstrate the effectiveness of our approach in practice.

Live content is unavailable. Log in and register to view live content