Timezone: »

 
Oral
Shampoo: Preconditioned Stochastic Tensor Optimization
Vineet Gupta · Tomer Koren · Yoram Singer

Thu Jul 12 04:30 AM -- 04:50 AM (PDT) @ A9

Preconditioned gradient methods are among the most general and powerful toolsin optimization. However, preconditioning requires storing and manipulatingprohibitively large matrices. We describe and analyze a new structure-awarepreconditioning algorithm, called Shampoo, for stochastic optimization overtensor spaces. Shampoo maintains a set of preconditioning matrices, each ofwhich operates on a single dimension, contracting over the remainingdimensions. We establish convergence guarantees in the stochastic convexsetting, the proof of which builds upon matrix trace inequalities. Ourexperiments with state-of-the-art deep learning models show that Shampoo iscapable of converging considerably faster than commonly used optimizers.Surprisingly, although it involves a more complex update rule, Shampoo's runtime per step is comparable in practice to that of simple gradientmethods such as SGD, AdaGrad, and Adam.

Author Information

Vineet Gupta (Google)
Tomer Koren (Google Brain)
Yoram Singer (Google)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors