EEG-FM-Bench: A Comprehensive Benchmark for the Systematic Evaluation and Diagnostic Analyses of EEG Foundation Models

Electroencephalography foundation models (EEG-FMs) have advanced brain signal analysis, but the lack of standardized evaluation benchmarks impedes model comparison and scientific progress. Current evaluations rely on inconsistent protocols that render cross-model comparisons unreliable, while a lack of diagnostic analyses obscures the internal mechanisms driving transfer efficiency and scaling behaviors. To address this, we introduce EEG-FM-Bench, a unified system for the standardized evaluation of EEG-FMs. The benchmark integrates 14 datasets across 10 paradigms and incorporates diverse experimental settings, including multiple fine-tuning strategies, task organizations, and classifier configurations, supported by tools for gradient and representation analysis. Our experiments and analysis reveal several critical insights: (1) multi-task learning often acts as a useful regularizer that mitigates overfitting in data-scarce EEG contexts, although negative transfer can arise under specific task paradigms; (2) pre-training efficiency is currently limited by gradient conflicts between reconstruction objectives and downstream tasks; (3) under released checkpoints and a matched downstream protocol, model or data scale alone does not fully explain transfer performance, while objective alignment, adaptation compatibility, and EEG-specific design appear to be important factors. This benchmark enables fair comparison and reproducible analysis, providing a step toward fairer comparison and more interpretable analysis of EEG-FMs. Code is available at https://github.com/xw1216/EEG-FM-Bench.

Brain-wave recordings (EEG) can help us study sleep, emotions, movement, and disease, but they are noisy and often hard to label. Recently, researchers have built large pre-trained EEG models, hoping they can be reused across many brain-signal tasks. However, these models are usually tested with different datasets, preprocessing steps, and training rules, making it hard to know which models truly work better or why. We introduce EEG-FM-Bench, a unified benchmark for evaluating EEG foundation models under the same conditions. It brings together 14 public datasets covering 10 common EEG applications, including motor imagery, sleep staging, emotion recognition, seizure detection, and disease diagnosis. The benchmark tests multiple ways of adapting models to new tasks and includes tools that look inside the models by analyzing their learning signals and internal representations. Our results show that joint training across tasks can often make models more robust, but unrelated tasks can sometimes interfere with each other. We also find that using more pre-training data or larger models does not guarantee better performance, because the pre-training objective may not match what downstream tasks need. EEG-FM-Bench therefore gives researchers a fairer way to compare models and a clearer guide for designing future EEG AI systems.