IBMA: Information Bottleneck-Based Multimodal Alignment

Multimodal learning aims to integrate information from heterogeneous data sources to improve representation quality and downstream task performance. A key challenge lies in aligning modality-specific representations while suppressing modality-dependent noise and redundancy. The Information Bottleneck (IB) principle provides a principled framework for learning task-relevant representations. Existing multimodal IB methods primarily apply the IB principle to fused multimodal representation and rely on restrictive distributional assumptions, such as Gaussian latent priors induced by variational autoencoders, which may not hold in practice. In this paper, we propose Information Bottleneck–based Multimodal Alignment (IBMA), a novel multimodal learning framework that enforces the IB principle for both the fused multimodal representation and modality-specific representations. IBMA introduces modality-specific representation alignment that guides each modality-specific encoder to learn informative and task-relevant representations aligned with the complementary modality, thereby enhancing cross-modal semantic consistency. Moreover, we derive a novel, efficient, and distribution-free variational upper bound for the IB loss that avoids unrealistic assumptions on latent feature distributions and is readily optimized using standard stochastic gradient descent. Extensive experiments demonstrate that IBMA achieves superior performance compared to existing multimodal learning methods, validating the effectiveness of modality-specific representation alignment. The code for IBMA is available at https://github.com/Statistical-Deep-Learning/IBMA.

Many real-world AI systems need to understand information from multiple sources at the same time. For example, a medical diagnosis system may use both a chest X-ray and a written clinical report, while an emotion recognition system may use facial expressions, speech, and text. These different sources can provide complementary clues, but they can also contain noise, irrelevant details, or information that does not match well across sources. This makes it difficult for AI models to combine them reliably. In this paper, we propose IBMA, a new method for helping AI models better learn from multiple types of data. The key idea is to encourage each data source to keep the information that is useful for the final prediction while reducing distracting or source-specific details. At the same time, IBMA encourages different sources to guide each other, so that the model learns representations that are more consistent across modalities. For instance, in emotion recognition, spoken words can help the model focus on the most relevant facial expressions; in medical image analysis, a clinical report can help guide attention to important regions in an X-ray. Unlike some earlier methods, IBMA does not rely on strong assumptions about how the model’s internal features are distributed, making it more flexible for complex real-world data. Experiments on several multimodal tasks, including emotion recognition, sentiment analysis, tissue analysis, and chest disease classification, show that IBMA performs better than existing methods. These results suggest that improving how different data sources align with and support each other can make multimodal AI systems more accurate and reliable.