Timezone: »

Tractable structured natural-gradient descent using local parameterizations
Wu Lin · Frank Nielsen · Khan Emtiyaz · Mark Schmidt

Wed Jul 21 06:35 AM -- 06:40 AM (PDT) @ None

Natural-gradient descent (NGD) on structured parameter spaces (e.g., low-rank covariances) is computationally challenging due to difficult Fisher-matrix computations. We address this issue by using \emph{local-parameter coordinates} to obtain a flexible and efficient NGD method that works well for a wide-variety of structured parameterizations. We show four applications where our method (1) generalizes the exponential natural evolutionary strategy, (2) recovers existing Newton-like algorithms, (3) yields new structured second-order algorithms, and (4) gives new algorithms to learn covariances of Gaussian and Wishart-based distributions. We show results on a range of problems from deep learning, variational inference, and evolution strategies. Our work opens a new direction for scalable structured geometric methods.

Author Information

Wu Lin (University of British Columbia)
Frank Nielsen (Sony CSL, Japan)
Khan Emtiyaz (RIKEN)
Mark Schmidt (University of British Columbia)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors