The aim of this work is to enable inference of deep networks that retain high accuracy for the least possible model complexity, with the latter deduced from the data during inference. To this end, we revisit deep networks that comprise competing linear units, as opposed to nonlinear units that do not entail any form of (local) competition. In this context, our main technical innovation consists in an inferential setup that leverages solid arguments from Bayesian nonparametrics. We infer both the needed set of connections or locally competing sets of units, as well as the required floating-point precision for storing the network parameters. Specifically, we introduce auxiliary discrete latent variables representing which initial network components are actually needed for modeling the data at hand, and perform Bayesian inference over them by imposing appropriate stick-breaking priors. As we experimentally show using benchmark datasets, our approach yields networks with less computational footprint than the state-of-the-art, and with no compromises in predictive accuracy.
Konstantinos Panousis (National and Kapodistrian University of Athens)
Sotirios Chatzis (Cyprus University of Technology)
Dr. Sotirios P. Chatzis, (male) is an Assistant Professor at the Department of Electrical Engineering, Computer Engineering and Informatics (EECEI), Cyprus University of Technology. He has first-authored more than 60 papers in the most prestigious journals and conferences of his research field. He is PC member of several top-tier venues in Machine Learning, including NIPS and ICML. He is serving as an Associate Editor of the IEEE Transactions on Signal Processing. His major research interests revolve around the themes of approximate (variational) inference, deep generative models, and Bayesian nonparametrics.
Sergios Theodoridis (National and Kapodistrian University of Athens)
Related Events (a corresponding poster, oral, or spotlight)
2019 Oral: Nonparametric Bayesian Deep Networks with Local Competition »
Thu Jun 13th 04:20 -- 04:25 PM Room Grand Ballroom