Timezone: »
Oral
Efficient Full-Matrix Adaptive Regularization
Naman Agarwal · Brian Bullins · Xinyi Chen · Elad Hazan · Karan Singh · Cyril Zhang · Yi Zhang
Adaptive regularization methods pre-multiply a descent direction by a preconditioning matrix. Due to the large number of parameters of machine learning problems, full-matrix preconditioning methods are prohibitively expensive. We show how to modify full-matrix adaptive regularization in order to make it practical and effective. We also provide novel theoretical analysis for adaptive regularization in non-convex optimization settings. The core of our algorithm, termed GGT, consists of efficient inverse computation of square roots of low-rank matrices. Our preliminary experiments underscore improved convergence rate of GGT across a variety of synthetic tasks and standard deep learning benchmarks.
Author Information
Naman Agarwal (Google AI Princeton)
Brian Bullins (Princeton University)
Xinyi Chen (Google Research)
Elad Hazan (Princeton University)
Karan Singh (Princeton University)
Cyril Zhang (Princeton University)
Yi Zhang (Princeton University)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: Efficient Full-Matrix Adaptive Regularization »
Fri. Jun 14th 01:30 -- 04:00 AM Room Pacific Ballroom #209
More from the Same Authors
-
2021 : Robust online control with model misspecification »
Xinyi Chen · Udaya Ghai · Elad Hazan · Alexandre Megretsky -
2021 : A Boosting Approach to Reinforcement Learning »
Nataly Brukhim · Elad Hazan · Karan Singh -
2022 : Non-convex online learning via algorithmic equivalence »
Udaya Ghai · Zhou Lu · Elad Hazan -
2023 Poster: Multi-User Reinforcement Learning with Low Rank Rewards »
Dheeraj Nagaraj · Suhas Kowshik · Naman Agarwal · Praneeth Netrapalli · Prateek Jain -
2022 Poster: A Regret Minimization Approach to Multi-Agent Control »
Udaya Ghai · Udari Madhuhshani · Naomi Leonard · Elad Hazan -
2022 Spotlight: A Regret Minimization Approach to Multi-Agent Control »
Udaya Ghai · Udari Madhuhshani · Naomi Leonard · Elad Hazan -
2021 Poster: Boosting for Online Convex Optimization »
Elad Hazan · Karan Singh -
2021 Spotlight: Boosting for Online Convex Optimization »
Elad Hazan · Karan Singh -
2021 Poster: A Regret Minimization Approach to Iterative Learning Control »
Naman Agarwal · Elad Hazan · Anirudha Majumdar · Karan Singh -
2021 Spotlight: A Regret Minimization Approach to Iterative Learning Control »
Naman Agarwal · Elad Hazan · Anirudha Majumdar · Karan Singh -
2021 Poster: Acceleration via Fractal Learning Rate Schedules »
Naman Agarwal · Surbhi Goel · Cyril Zhang -
2021 Spotlight: Acceleration via Fractal Learning Rate Schedules »
Naman Agarwal · Surbhi Goel · Cyril Zhang -
2021 Tutorial: Online and non-stochastic control »
Elad Hazan · Karan Singh -
2021 : Online and non-stochastic control »
Elad Hazan -
2020 Poster: Calibration, Entropy Rates, and Memory in Language Models »
Mark Braverman · Xinyi Chen · Sham Kakade · Karthik Narasimhan · Cyril Zhang · Yi Zhang -
2020 Poster: A Sample Complexity Separation between Non-Convex and Convex Meta-Learning »
Nikunj Umesh Saunshi · Yi Zhang · Mikhail Khodak · Sanjeev Arora -
2020 Poster: Boosting for Control of Dynamical Systems »
Naman Agarwal · Nataly Brukhim · Elad Hazan · Zhou Lu -
2019 Poster: Online Control with Adversarial Disturbances »
Naman Agarwal · Brian Bullins · Elad Hazan · Sham Kakade · Karan Singh -
2019 Oral: Online Control with Adversarial Disturbances »
Naman Agarwal · Brian Bullins · Elad Hazan · Sham Kakade · Karan Singh -
2019 Poster: Provably Efficient Maximum Entropy Exploration »
Elad Hazan · Sham Kakade · Karan Singh · Abby Van Soest -
2019 Oral: Provably Efficient Maximum Entropy Exploration »
Elad Hazan · Sham Kakade · Karan Singh · Abby Van Soest -
2018 Poster: Stronger Generalization Bounds for Deep Nets via a Compression Approach »
Sanjeev Arora · Rong Ge · Behnam Neyshabur · Yi Zhang -
2018 Oral: Stronger Generalization Bounds for Deep Nets via a Compression Approach »
Sanjeev Arora · Rong Ge · Behnam Neyshabur · Yi Zhang -
2018 Poster: On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization »
Sanjeev Arora · Nadav Cohen · Elad Hazan -
2018 Oral: On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization »
Sanjeev Arora · Nadav Cohen · Elad Hazan -
2017 Poster: The Price of Differential Privacy For Online Learning »
Naman Agarwal · Karan Singh -
2017 Talk: The Price of Differential Privacy For Online Learning »
Naman Agarwal · Karan Singh -
2017 Poster: Efficient Regret Minimization in Non-Convex Games »
Elad Hazan · Karan Singh · Cyril Zhang -
2017 Talk: Efficient Regret Minimization in Non-Convex Games »
Elad Hazan · Karan Singh · Cyril Zhang