Timezone: »

 
Spotlight
Decomposed Mutual Information Estimation for Contrastive Representation Learning
Alessandro Sordoni · Nouha Dziri · Hannes Schulz · Geoff Gordon · Philip Bachman · Remi Tachet des Combes

Thu Jul 22 05:40 PM -- 05:45 PM (PDT) @
in Representation Learning 4 »

Recent contrastive representation learning methods rely on estimating mutual information (MI) between multiple views of an underlying context. E.g., we can derive multiple views of a given image by applying data augmentation, or we can split a sequence into views comprising the past and future of some step in the sequence. Contrastive lower bounds on MI are easy to optimize, but have a strong underestimation bias when estimating large amounts of MI. We propose decomposing the full MI estimation problem into a sum of smaller estimation problems by splitting one of the views into progressively more informed subviews and by applying the chain rule on MI between the decomposed views. This expression contains a sum of unconditional and conditional MI terms, each measuring modest chunks of the total MI, which facilitates approximation via contrastive bounds. To maximize the sum, we formulate a contrastive lower bound on the conditional MI which can be approximated efficiently. We refer to our general approach as Decomposed Estimation of Mutual Information (DEMI). We show that DEMI can capture a larger amount of MI than standard non-decomposed contrastive bounds in a synthetic setting, and learns better representations in a vision domain and for dialogue generation.

Keywords: Unsupervised Learning · Algorithms

Author Information

Alessandro Sordoni (Microsoft Research)
Nouha Dziri (University of Alberta)

I’m a PhD student at the University of Alberta where I investigate generative deep learning models and natural language processing methods. In particular, my research focuses on developing data-driven approaches for computational natural language understanding, primarily in the context of enabling machines to converse with humans in natural language. Further, I’m interested in exploring different methods for the fiendishly difficult problem of evaluating conversational AI.

Hannes Schulz (Microsoft)
Geoff Gordon (Carnegie Mellon University)
Philip Bachman (Microsoft Research)
Remi Tachet des Combes (Microsoft Research Montreal)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors