Timezone: »

 
Poster
A Free-Energy Principle for Representation Learning
Yansong Gao · Pratik Chaudhari

Tue Jul 14 07:00 AM -- 07:45 AM & Tue Jul 14 06:00 PM -- 06:45 PM (PDT) @
in Poster Session 1 »

This paper employs a formal connection of machine learning with thermodynamics to characterize the quality of learnt representations for transfer learning. We discuss how information-theoretic functionals such as rate, distortion and classification loss of a model lie on a convex, so-called equilibrium surface. We prescribe dynamical processes to traverse this surface under constraints, e.g., an iso-classification process that trades off rate and distortion to keep the classification loss unchanged. We demonstrate how this process can be used for transferring representations from a source dataset to a target dataset while keeping the classification loss constant. Experimental validation of the theoretical results is provided on standard image-classification datasets.

Keywords: Bayesian Deep Learning · Information Theory and Estimation · Representation Learning · Transfer and Multitask Learning

Author Information

Yansong Gao (University of Pennsylvania)
Pratik Chaudhari (University of Pennsylvania)

More from the Same Authors