Timezone: »
There has been considerable recent interest in mutual information (MI) minimization for various machine learning tasks. However, estimating and minimizing MI in high-dimensional spaces remains a challenging problem, especially when only samples are accessible, rather than the underlying distribution forms. Previous works mainly focus on MI lower bound approximation, which is not applicable to MI minimization problems. In this paper, we propose a novel Contrastive Log-ratio Upper Bound (CLUB) of mutual information. We provide a theoretical analysis of the properties of CLUB and its variational approximation. Based on this upper bound, we introduce an accelerated MI minimization training scheme, that bridges MI minimization with negative sampling. Simulation studies on Gaussian distributions show that CLUB provides reliable estimates. Real-world MI minimization experiments, including domain adaptation and the information bottleneck, further demonstrate the effectiveness of the proposed method.
Author Information
Pengyu Cheng (Duke University)
Weituo Hao (Duke University)
Shuyang Dai (Duke University)
Jiachang Liu (Duke University)
Zhe Gan (Microsoft)
Lawrence Carin (Duke)
More from the Same Authors
-
2021 : Hölder Bounds for Sensitivity Analysis in Causal Reasoning »
Serge Assaad · Shuxi Zeng · Henry Pfister · Fan Li · Lawrence Carin -
2020 Poster: Learning Autoencoders with Relational Regularization »
Hongteng Xu · Dixin Luo · Ricardo Henao · Svati Shah · Lawrence Carin -
2020 Poster: Graph Optimal Transport for Cross-Domain Alignment »
Liqun Chen · Zhe Gan · Yu Cheng · Linjie Li · Lawrence Carin · Jingjing Liu -
2020 Poster: On Leveraging Pretrained GANs for Generation with Limited Data »
Miaoyun Zhao · Yulai Cong · Lawrence Carin -
2019 Poster: Gromov-Wasserstein Learning for Graph Matching and Node Embedding »
Hongteng Xu · Dixin Luo · Hongyuan Zha · Lawrence Carin -
2019 Oral: Gromov-Wasserstein Learning for Graph Matching and Node Embedding »
Hongteng Xu · Dixin Luo · Hongyuan Zha · Lawrence Carin -
2019 Poster: Stochastic Blockmodels meet Graph Neural Networks »
Nikhil Mehta · Lawrence Carin · Piyush Rai -
2019 Poster: Understanding and Accelerating Particle-Based Variational Inference »
Chang Liu · Jingwei Zhuo · Pengyu Cheng · RUIYI (ROY) ZHANG · Jun Zhu -
2019 Poster: Variational Annealing of GANs: A Langevin Perspective »
Chenyang Tao · Shuyang Dai · Liqun Chen · Ke Bai · Junya Chen · Chang Liu · RUIYI (ROY) ZHANG · Georgiy Bobashev · Lawrence Carin -
2019 Oral: Stochastic Blockmodels meet Graph Neural Networks »
Nikhil Mehta · Lawrence Carin · Piyush Rai -
2019 Oral: Understanding and Accelerating Particle-Based Variational Inference »
Chang Liu · Jingwei Zhuo · Pengyu Cheng · RUIYI (ROY) ZHANG · Jun Zhu -
2019 Oral: Variational Annealing of GANs: A Langevin Perspective »
Chenyang Tao · Shuyang Dai · Liqun Chen · Ke Bai · Junya Chen · Chang Liu · RUIYI (ROY) ZHANG · Georgiy Bobashev · Lawrence Carin -
2018 Poster: Learning Registered Point Processes from Idiosyncratic Observations »
Hongteng Xu · Lawrence Carin · Hongyuan Zha -
2018 Poster: Policy Optimization as Wasserstein Gradient Flows »
RUIYI (ROY) ZHANG · Changyou Chen · Chunyuan Li · Lawrence Carin -
2018 Poster: JointGAN: Multi-Domain Joint Distribution Learning with Generative Adversarial Nets »
Yunchen Pu · Shuyang Dai · Zhe Gan · Weiyao Wang · Guoyin Wang · Yizhe Zhang · Ricardo Henao · Lawrence Carin -
2018 Oral: Policy Optimization as Wasserstein Gradient Flows »
RUIYI (ROY) ZHANG · Changyou Chen · Chunyuan Li · Lawrence Carin -
2018 Oral: JointGAN: Multi-Domain Joint Distribution Learning with Generative Adversarial Nets »
Yunchen Pu · Shuyang Dai · Zhe Gan · Weiyao Wang · Guoyin Wang · Yizhe Zhang · Ricardo Henao · Lawrence Carin -
2018 Oral: Learning Registered Point Processes from Idiosyncratic Observations »
Hongteng Xu · Lawrence Carin · Hongyuan Zha -
2018 Poster: Adversarial Time-to-Event Modeling »
Paidamoyo Chapfuwa · Chenyang Tao · Chunyuan Li · Courtney Page · Benjamin Goldstein · Lawrence Carin · Ricardo Henao -
2018 Oral: Adversarial Time-to-Event Modeling »
Paidamoyo Chapfuwa · Chenyang Tao · Chunyuan Li · Courtney Page · Benjamin Goldstein · Lawrence Carin · Ricardo Henao -
2018 Poster: Continuous-Time Flows for Efficient Inference and Density Estimation »
Changyou Chen · Chunyuan Li · Liquan Chen · Wenlin Wang · Yunchen Pu · Lawrence Carin -
2018 Poster: Chi-square Generative Adversarial Network »
Chenyang Tao · Liqun Chen · Ricardo Henao · Jianfeng Feng · Lawrence Carin -
2018 Poster: Variational Inference and Model Selection with Generalized Evidence Bounds »
Liqun Chen · Chenyang Tao · RUIYI (ROY) ZHANG · Ricardo Henao · Lawrence Carin -
2018 Oral: Chi-square Generative Adversarial Network »
Chenyang Tao · Liqun Chen · Ricardo Henao · Jianfeng Feng · Lawrence Carin -
2018 Oral: Continuous-Time Flows for Efficient Inference and Density Estimation »
Changyou Chen · Chunyuan Li · Liquan Chen · Wenlin Wang · Yunchen Pu · Lawrence Carin -
2018 Oral: Variational Inference and Model Selection with Generalized Evidence Bounds »
Liqun Chen · Chenyang Tao · RUIYI (ROY) ZHANG · Ricardo Henao · Lawrence Carin -
2017 Poster: Stochastic Gradient Monomial Gamma Sampler »
Yizhe Zhang · Changyou Chen · Zhe Gan · Ricardo Henao · Lawrence Carin -
2017 Poster: Adversarial Feature Matching for Text Generation »
Yizhe Zhang · Zhe Gan · Kai Fan · Zhi Chen · Ricardo Henao · Dinghan Shen · Lawrence Carin -
2017 Talk: Adversarial Feature Matching for Text Generation »
Yizhe Zhang · Zhe Gan · Kai Fan · Zhi Chen · Ricardo Henao · Dinghan Shen · Lawrence Carin -
2017 Talk: Stochastic Gradient Monomial Gamma Sampler »
Yizhe Zhang · Changyou Chen · Zhe Gan · Ricardo Henao · Lawrence Carin -
2017 Poster: Deep Generative Models for Relational Data with Side Information »
Changwei Hu · Piyush Rai · Lawrence Carin -
2017 Talk: Deep Generative Models for Relational Data with Side Information »
Changwei Hu · Piyush Rai · Lawrence Carin