Timezone: »

 
Oral
Learning-to-Learn Stochastic Gradient Descent with Biased Regularization
Giulia Denevi · Carlo Ciliberto · Riccardo Grazzi · Massimiliano Pontil

Wed Jun 12 03:05 PM -- 03:10 PM (PDT) @ Room 201
in Transfer and Multitask Learning »

We study the problem of learning-to-learn: inferring a learning algorithm that works well on tasks sampled from an unknown distribution. As class of algorithms we consider Stochastic Gradient Descent on the true risk regularized by the square euclidean distance to a bias vector. We present an average excess risk bound for such a learning algorithm. This result quantifies the potential benefit of using a bias vector with respect to the unbiased case. We then address the problem of estimating the bias from a sequence of tasks. We propose a meta-algorithm which incrementally updates the bias, as new tasks are observed. The low space and time complexity of this approach makes it appealing in practice. We provide guarantees on the learning ability of the meta-algorithm. A key feature of our results is that, when the number of tasks grows and their variance is relatively small, our learning-to-learn approach has a significant advantage over learning each task in isolation by Stochastic Gradient Descent without a bias term. We report on numerical experiments which demonstrate the effectiveness of our approach.

[ Slides]  [ Video] 

Author Information

Giulia Denevi (IIT)
Carlo Ciliberto (Imperial College London)
Riccardo Grazzi (Istituto Italiano di Tecnologia - University College London)
Massimiliano Pontil (University College London)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors