Timezone: »

 
Oral
Shampoo: Preconditioned Stochastic Tensor Optimization
Vineet Gupta · Tomer Koren · Yoram Singer

Thu Jul 12 04:30 AM -- 04:50 AM (PDT) @ A9
in Optimization (Convex) 3 »

Preconditioned gradient methods are among the most general and powerful toolsin optimization. However, preconditioning requires storing and manipulatingprohibitively large matrices. We describe and analyze a new structure-awarepreconditioning algorithm, called Shampoo, for stochastic optimization overtensor spaces. Shampoo maintains a set of preconditioning matrices, each ofwhich operates on a single dimension, contracting over the remainingdimensions. We establish convergence guarantees in the stochastic convexsetting, the proof of which builds upon matrix trace inequalities. Ourexperiments with state-of-the-art deep learning models show that Shampoo iscapable of converging considerably faster than commonly used optimizers.Surprisingly, although it involves a more complex update rule, Shampoo's runtime per step is comparable in practice to that of simple gradientmethods such as SGD, AdaGrad, and Adam.

[ PDF] 

Author Information

Vineet Gupta (Google)
Tomer Koren (Google Brain)
Yoram Singer (Google)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors