EEG-Based Multimodal Learning via Hyperbolic Mixture-of-Curvature Experts

Electroencephalography (EEG)-based multimodal learning integrates brain signals with complementary modalities to improve mental state assessment, providing great clinical potential. The effectiveness of such paradigms largely depends on the representation learning on heterogeneous modalities. For EEG-based paradigms, one promising approach is to leverage their hierarchical structures, as recent studies have shown that both EEG and associated modalities (e.g., facial expressions) exhibit hierarchical structures reflecting complex cognitive processes. However, Euclidean embeddings struggle to represent these hierarchical structures due to their flat geometry, while hyperbolic spaces, with their exponential growth property, are naturally suited for them. In this work, we propose EEG-MoCE, a novel hyperbolic mixture-of-curvature experts framework designed for multimodal neurotechnology. EEG-MoCE assigns each modality to an expert in a learnable-curvature hyperbolic space, enabling adaptive modeling of its intrinsic geometry. A curvature-aware fusion strategy then dynamically weights experts, emphasizing modalities with richer hierarchical information. Extensive experiments on benchmark datasets demonstrate that EEG-MoCE achieves state-of-the-art performance, including emotion recognition, sleep staging, and cognitive assessment. Code is available at https://github.com/zhourunhe/EEG-MoCE.

Decoding a person’s mental state from brain recordings is challenging because brain signals are noisy and vary across people. In many real-world settings, brain recordings are combined with other information, such as facial expressions and speech, to better understand emotions, sleep states, or mental workload. However, these different types of signals do not always carry information in the same way. Some contain simple patterns, while others contain more hierarchical relationships. We developed EEG-MoCE, a framework that gives each type of signal its own specialist model. Each specialist learns how much hierarchical structure is useful for its signal, instead of forcing all signals into the same representation. The framework then combines the specialists by placing more emphasis on the signals that provide more useful structure. We tested EEG-MoCE on three mental-state recognition problems. Across these tasks, EEG-MoCE performed better than strong existing methods. These results suggest that modeling different signals in more flexible ways can make mental-state monitoring more reliable, with potential uses in healthcare, education, and human computer interaction.