Timezone: »

 
Spotlight
On Implicit Bias in Overparameterized Bilevel Optimization
Paul Vicol · Jonathan Lorraine · Fabian Pedregosa · David Duvenaud · Roger Grosse

Tue Jul 19 11:50 AM -- 11:55 AM (PDT) @ Room 318 - 320
in Optimization »

Many problems in machine learning involve bilevel optimization (BLO), including hyperparameter optimization, meta-learning, and dataset distillation. Bilevel problems involve inner and outer parameters, each optimized for its own objective. Often, at least one of the two levels is underspecified and there are multiple ways to choose among equivalent optima. Inspired by recent studies of the implicit bias induced by optimization algorithms in single-level optimization, we investigate the implicit bias of different gradient-based algorithms for jointly optimizing the inner and outer parameters. We delineate two standard BLO methods---cold-start and warm-start BLO---and show that the converged solution or long-run behavior depends to a large degree on these and other algorithmic choices, such as the hypergradient approximation. We also show that the solutions from warm-start BLO can encode a surprising amount of information about the outer objective, even when the outer optimization variables are low-dimensional. We believe that implicit bias deserves as central a role in the study of bilevel optimization as it has attained in the study of single-level neural net optimization.

Keywords: Optimization · OPT: Bilevel optimization · OPT: Higher order

Author Information

Paul Vicol (University of Toronto)
Jonathan Lorraine (University of Toronto)
Fabian Pedregosa (Google)
David Duvenaud (University of Toronto)
Roger Grosse (University of Toronto and Vector Institute)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors