Timezone: »

 
Poster
CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information
Pengyu Cheng · Weituo Hao · Shuyang Dai · Jiachang Liu · Zhe Gan · Lawrence Carin

Thu Jul 16 06:00 AM -- 06:45 AM & Thu Jul 16 05:00 PM -- 05:45 PM (PDT) @
in Poster Session 41 »

There has been considerable recent interest in mutual information (MI) minimization for various machine learning tasks. However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples are accessible, rather than the underlying distribution forms. Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems. In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation. Based on this upper bound, we introduce an accelerated MI minimization training scheme, that bridges MI minimization with negative sampling. Simulation studies on Gaussian distributions show that CLUB provides reliable estimates. Real-world MI minimization experiments, including domain adaptation and the information bottleneck, further demonstrate the effectiveness of the proposed method.

Keywords: Bayesian Deep Learning · Deep Learning - General

Author Information

Pengyu Cheng (Duke University)
Weituo Hao (Duke University)
Shuyang Dai (Duke University)
Jiachang Liu (Duke University)
Zhe Gan (Microsoft)
Lawrence Carin (Duke)

More from the Same Authors