Rethinking Gating Mechanism in Sparse MoE: Handling Arbitrary Modality Inputs with Confidence-Guided Gate

Effectively managing missing modalities is a fundamental challenge in real-world multimodal learning scenarios, where data incompleteness often results from systematic collection errors or sensor failures. Sparse Mixture-of-Experts (SMoE) architectures have the potential to naturally handle multimodal data, with individual experts specializing in different modalities. However, existing SMoE approach often lacks proper ability to handle missing modality, leading to performance degradation and poor generalization in real-world applications. We propose ConfSMoE to introduce a two-stage imputation module to handle the missing modality problem for the SMoE architecture by taking the opinion of experts and reveal the insight of expert collapse from theoretical analysis with strong empirical evidence. Inspired by our theoretical analysis, ConfSMoE propose a novel expert gating mechanism by detaching the softmax routing score to task confidence score w.r.t ground truth signal. This naturally relieves expert collapse without introducing additional load balance loss function. We show that the insights of expert collapse aligns with other gating mechanism such as Gaussian and Laplacian gate. The proposed method is evaluated on four different real world dataset with three distinct experiment settings to conduct comprehensive analysis of ConfSMoE on resistance to missing modality and the impacts of proposed gating mechanism.

Modern AI systems are designed to process multiple types of data at once, like video, audio, and text, to make complex decisions. However, in real world scenarios, data is rarely perfect; sensors fail or information gets lost, leaving the AI with incomplete puzzles. When this happens, a popular AI architecture called Mixture of Experts which splits tasks among specialized artificial experts gets easily confused. We show that this missing data causes a glitch called expert collapse, where the system stubbornly over relies on just one or two experts while the rest sit completely idle. In this paper, we explore the hidden mathematical root causes of this breakdown and introduce a novel framework called ConfSMoE. By fundamentally rethinking how the MoE handles missing information and chooses which experts to activate, our method keeps all experts working in harmony. This results in a significantly more reliable and robust system, even when large chunks of data are missing.