Oral
Efficient Full-Matrix Adaptive Regularization
Naman Agarwal · Brian Bullins · Xinyi Chen · Elad Hazan · Karan Singh · Cyril Zhang · Yi Zhang

Thu Jun 13th 11:25 -- 11:30 AM @ Room 101
in Optimization and Graphical Models »

Adaptive regularization methods pre-multiply a descent direction by a preconditioning matrix. Due to the large number of parameters of machine learning problems, full-matrix preconditioning methods are prohibitively expensive. We show how to modify full-matrix adaptive regularization in order to make it practical and effective. We also provide novel theoretical analysis for adaptive regularization in non-convex optimization settings. The core of our algorithm, termed GGT, consists of efficient inverse computation of square roots of low-rank matrices. Our preliminary experiments underscore improved convergence rate of GGT across a variety of synthetic tasks and standard deep learning benchmarks.

Author Information

Naman Agarwal (Google AI Princeton)
Brian Bullins (Princeton University)
Xinyi Chen (Google Research)
Elad Hazan (Princeton University)
Karan Singh (Princeton University)
Cyril Zhang (Princeton University)
Yi Zhang (Princeton University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors