Timezone: »

 
Poster
Deep Reference Priors: What is the best way to pretrain a model?
Yansong Gao · Rahul Ramesh · Pratik Chaudhari

Wed Jul 20 03:30 PM -- 05:30 PM (PDT) @ Hall E #711
in Poster Session 2 »

What is the best way to exploit extra data -- be it unlabeled data from the same task, or labeled data from a related task -- to learn a given task? This paper formalizes the question using the theory of reference priors. Reference priors are objective, uninformative Bayesian priors that maximize the mutual information between the task and the weights of the model. Such priors enable the task to maximally affect the Bayesian posterior, e.g., reference priors depend upon the number of samples available for learning the task and for very small sample sizes, the prior puts more probability mass on low-complexity models in the hypothesis space. This paper presents the first demonstration of reference priors for medium-scale deep networks and image-based data. We develop generalizations of reference priors and demonstrate applications to two problems. First, by using unlabeled data to compute the reference prior, we develop new Bayesian semi-supervised learning methods that remain effective even with very few samples per class. Second, by using labeled data from the source task to compute the reference prior, we develop a new pretraining method for transfer learning that allows data from the target task to maximally affect the Bayesian posterior. Empirical validation of these methods is conducted on image classification datasets. Code is available at https://github.com/grasp-lyrl/deepreferencepriors

Keywords: PM: Bayesian Models and Methods · PM: Variational Inference

Author Information

Yansong Gao (University of Pennsylvania)
Rahul Ramesh (University of Pennsylvania)
Pratik Chaudhari (University of Pennsylvania, Amazon Web Services)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors