Hybrid Models with Deep and Invertible Features

Deep neural networks are powerful black-box predictors for modeling conditional distributions of the form p(target|features). While they can be very successful at supervised learning problems where the train and test distributions are the same, they can make overconfident wrong predictions when the test distribution is different. Hybrid models that include both a discriminative conditional model p(target|features) and a generative model p(features) can be more robust under dataset shift, as they can detect covariate shift using the generative model. Current state-of-the-art hybrid models require approximate inference which can be computationally expensive. We propose an hybrid model that defines a generalized linear model on top of deep invertible features (e.g. normalizing flows). An attractive property of our model is that both p(features), the log density, and p(target|features), the predictive distribution, can be computed exactly in a single feed-forward pass. We show that our hybrid model achieves similar predictive accuracy as purely discriminative models on classification and regression tasks, while providing better uncertainty quantification and the ability to detect out-of-distribution inputs. In addition, we also demonstrate that the generative component of the hybrid model can leverage unlabeled data for semi-supervised learning, as well as generate samples which are useful to visualize and interpret the model. The availability of the exact joint density p(target,features) also allows us to compute many quantities readily, making our hybrid model an useful building block for downstream applications of probabilistic deep learning, including but not limited to active learning and domain adaptation.