Timezone: »

 
Spotlight
Guarantees for Tuning the Step Size using a Learning-to-Learn Approach
Xiang Wang · Shuai Yuan · Chenwei Wu · Rong Ge

Wed Jul 21 07:30 AM -- 07:35 AM (PDT) @
in Learning Theory 6 »

Choosing the right parameters for optimization algorithms is often the key to their success in practice. Solving this problem using a learning-to-learn approach---using meta-gradient descent on a meta-objective based on the trajectory that the optimizer generates---was recently shown to be effective. However, the meta-optimization problem is difficult. In particular, the meta-gradient can often explode/vanish, and the learned optimizer may not have good generalization performance if the meta-objective is not chosen carefully. In this paper we give meta-optimization guarantees for the learning-to-learn approach on a simple problem of tuning the step size for quadratic loss. Our results show that the na\"ive objective suffers from meta-gradient explosion/vanishing problem. Although there is a way to design the meta-objective so that the meta-gradient remains polynomially bounded, computing the meta-gradient directly using backpropagation leads to numerical issues. We also characterize when it is necessary to compute the meta-objective on a separate validation set to ensure the generalization performance of the learned optimizer. Finally, we verify our results empirically and show that a similar phenomenon appears even for more complicated learned optimizers parametrized by neural networks.

Keywords: Computational Learning Theory

Author Information

Xiang Wang (Duke University)
Shuai Yuan (Duke University)
Chenwei Wu (Duke University)
Rong Ge (Duke University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors