Poster

Tractable structured natural-gradient descent using local parameterizations

Wu Lin · Frank Nielsen · Khan Emtiyaz · Mark Schmidt

Virtual

Keywords: [ Approximate Inference ]

[ Abstract ]
[ Slides
[ Paper ]
[ Visit Poster at Spot B1 in Virtual World ]
Wed 21 Jul 9 a.m. PDT — 11 a.m. PDT
 
Spotlight presentation: Optimization and Algorithms 3
Wed 21 Jul 6 a.m. PDT — 7 a.m. PDT

Abstract:

Natural-gradient descent (NGD) on structured parameter spaces (e.g., low-rank covariances) is computationally challenging due to difficult Fisher-matrix computations. We address this issue by using \emph{local-parameter coordinates} to obtain a flexible and efficient NGD method that works well for a wide-variety of structured parameterizations. We show four applications where our method (1) generalizes the exponential natural evolutionary strategy, (2) recovers existing Newton-like algorithms, (3) yields new structured second-order algorithms, and (4) gives new algorithms to learn covariances of Gaussian and Wishart-based distributions. We show results on a range of problems from deep learning, variational inference, and evolution strategies. Our work opens a new direction for scalable structured geometric methods.

Chat is not available.