Calibrated Multimodal Representation Learning with Missing Modalities

Multimodal representation learning harmonizes distinct modalities by aligning them into a unified latent space. Recent research generalizes traditional cross-modal alignment to produce enhanced multimodal synergy but requires all modalities to be present for a common instance, making it challenging to utilize prevalent datasets with missing modalities. We provide theoretical insights into this issue from an \textit{anchor shift} perspective. Observed modalities are aligned with a local anchor that deviates from the optimal one when all modalities are present, resulting in an inevitable shift. To address this, we propose CalMRL to calibrate incomplete alignments caused by missing modalities. CalMRL leverages the priors and the inherent connections among modalities to model the imputation for the missing ones at the representation level. To resolve the optimization dilemma, we employ a bi-step learning method with the closed-form solution of the posterior distribution of shared latents. We validate its mitigation of anchor shift and convergence with theoretical guidance. By equipping the calibrated alignment with the existing advanced method, we offer new flexibility to absorb data with missing modalities, which is originally unattainable. Extensive experiments demonstrate the superiority of CalMRL. The code is released at https://github.com/Xiaohao-Liu/CalMRL.

The real world is a mix of different modalities that work together to give us a complete picture. Models try to align these modalities into a shared understanding. However, real-world data is often messy and incomplete, with missing modalities. This creates a problem we call "anchor shift". We develop a calibration method called CalMRL. By modeling how different modalities relate to one another, CalMRL imputes the missing ones, corrects the bias, and keeps the alignment accurate. Our work explains why this drift happens and provides a practical way to fix it, allowing AI to learn more effectively from the imperfect, multi-sensory data found in the real world.