MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization

Deploying multimodal models in real-world scenarios requires generalization to new environments where recording conditions differ from training, a challenge known as multimodal domain generalization (MMDG). Standard architectures employ separate encoders for each modality and a fusion module, training the system end-to-end by optimizing on the fused features. In this paper, we identify that such joint optimization causes encoders to exploit cross-modal co-occurrences, statistical relationships between modalities that arise from source-specific recording conditions, rather than learning domain-invariant features. We term this failure mode Fusion Overfitting. To address this, we propose Modality-Entropy Regularization for Domain Generalization (MER-DG), which maximizes the entropy of each encoder's feature distribution to preserve feature diversity. MER-DG is architecture-agnostic and integrates into existing multimodal frameworks as an additive loss term. Extensive experiments on EPIC-Kitchens and HAC benchmarks demonstrate average improvements of ${\sim}5\%$ over standard fusion and ${\sim}2\%$ over state-of-the-art methods.

AI systems increasingly combine different types of data, such as video and audio, since each provides complementary information about what is happening. But these systems often fail when deployed in environments that differ from their training conditions, and we identified a specific reason why. When the video and audio parts of the model are trained jointly, each learns to lean on the other in ways tied to the training environment, a phenomenon we call Fusion Overfitting. Consider a model trained to recognize chopping in quiet home kitchens: the video stream learns to expect a particular sound pattern, and the audio stream a particular motion. This shortcut works at home but fails in a noisy commercial kitchen, where those patterns no longer line up. Our method, MER-DG, encourages each part of the model to keep its internal features diverse during training, so neither becomes overly reliant on the other. It works as a simple add-on to existing multimodal systems. Across two action-recognition benchmarks, MER-DG improves accuracy by about 5% on standard models and 2% on the best existing methods, with the largest gains when inputs are noisy or one stream is missing, situations common in real-world deployment.