Timezone: »

Variational Inference for Infinitely Deep Neural Networks
Achille Nazaret · David Blei

Wed Jul 20 11:20 AM -- 11:25 AM (PDT) @ Room 301 - 303
We introduce the unbounded depth neural network (UDN), a deep probabilistic model that adapts its complexity to the training data by using a neural network with infinite depth. The UDN contains an infinite sequence of hidden layers and places an unbounded prior on a truncation $\ell$, the layer from which it produces its data. Given a dataset of observations, the posterior UDN provides a conditional distribution of both the truncation and the parameters of the infinite neural network. We develop a variational inference algorithm that approximates this posterior, optimizing a distribution of neural network weights and of the truncation depth $\ell$, and without any upper limit on $\ell$. To this end, the variational family has a novel structure: It models neural network parameters of arbitrary depth, and it can dynamically create or remove free variational parameters to evolve its distribution of the truncation. With this family, gradient-based optimization of the ELBO naturally explores the space of truncations. We empirically study the UDN on real and synthetic data. UDN adapts its posterior depth to the dataset complexity; it outperforms standard neural networks of similar computational complexity; and it outperforms other approaches to infinite-depth neural networks.

Author Information

Achille Nazaret (Columbia University)
David Blei (Columbia University)

David Blei is a Professor of Statistics and Computer Science at Columbia University, and a member of the Columbia Data Science Institute. His research is in statistical machine learning, involving probabilistic topic models, Bayesian nonparametric methods, and approximate posterior inference algorithms for massive data. He works on a variety of applications, including text, images, music, social networks, user behavior, and scientific data. David has received several awards for his research, including a Sloan Fellowship (2010), Office of Naval Research Young Investigator Award (2011), Presidential Early Career Award for Scientists and Engineers (2011), Blavatnik Faculty Award (2013), and ACM-Infosys Foundation Award (2013). He is a fellow of the ACM.

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors