Timezone: »
Uncertainty computation in deep learning is essential to design robust and reliable systems. Variational inference (VI) is a promising approach for such computation, but requires more effort to implement and execute compared to maximum-likelihood methods. In this paper, we propose new natural-gradient algorithms to reduce such efforts for Gaussian mean-field VI. Our algorithms can be implemented within the Adam optimizer by perturbing the network weights during gradient evaluations, and uncertainty estimates can be cheaply obtained by using the vector that adapts the learning rate. This requires lower memory, computation, and implementation effort than existing VI methods, while obtaining uncertainty estimates of comparable quality. Our empirical results confirm this and further suggest that the weight-perturbation in our algorithm could be useful for exploration in reinforcement learning and stochastic optimization.
Author Information
Emti Khan (RIKEN)
Didrik Nielsen (RIKEN)
Voot Tangkaratt (RIKEN AIP)
Wu Lin (University of British Columbia)
Yarin Gal (University of OXford)
Akash Srivastava (MIT, IBM)
I’m a PhD student in the Informatics Forum, University of Edinburgh. I work with Dr Charles Sutton and Dr Michael U. Gutmann on variational inference for generative models using deep learning.
Related Events (a corresponding poster, oral, or spotlight)
-
2018 Oral: Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam »
Thu Jul 12th 02:50 -- 03:00 PM Room A4
More from the Same Authors
-
2021 Poster: Tractable structured natural-gradient descent using local parameterizations »
Wu Lin · Frank Nielsen · Khan Emtiyaz · Mark Schmidt -
2021 Spotlight: Tractable structured natural-gradient descent using local parameterizations »
Wu Lin · Frank Nielsen · Khan Emtiyaz · Mark Schmidt -
2020 Poster: Training Binary Neural Networks using the Bayesian Learning Rule »
Xiangming Meng · Roman Bachmann · Mohammad Emtiyaz Khan -
2020 Poster: Handling the Positive-Definite Constraint in the Bayesian Learning Rule »
Wu Lin · Mark Schmidt · Mohammad Emtiyaz Khan -
2020 Poster: Variational Imitation Learning with Diverse-quality Demonstrations »
Voot Tangkaratt · Bo Han · Mohammad Emtiyaz Khan · Masashi Sugiyama -
2019 Poster: Scalable Training of Inference Networks for Gaussian-Process Models »
Jiaxin Shi · Mohammad Emtiyaz Khan · Jun Zhu -
2019 Oral: Scalable Training of Inference Networks for Gaussian-Process Models »
Jiaxin Shi · Mohammad Emtiyaz Khan · Jun Zhu -
2019 Poster: Imitation Learning from Imperfect Demonstration »
Yueh-Hua Wu · Nontawat Charoenphakdee · Han Bao · Voot Tangkaratt · Masashi Sugiyama -
2019 Poster: Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations »
Wu Lin · Mohammad Emtiyaz Khan · Mark Schmidt -
2019 Oral: Imitation Learning from Imperfect Demonstration »
Yueh-Hua Wu · Nontawat Charoenphakdee · Han Bao · Voot Tangkaratt · Masashi Sugiyama -
2019 Oral: Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations »
Wu Lin · Mohammad Emtiyaz Khan · Mark Schmidt