MEG-GPT-2: A Generative Foundation Model for MEG with Structured Neural Representations

Foundation modelling of neural time series has largely focused on EEG and is typically evaluated with downstream tasks, leaving open questions about the fidelity of learnt representations. In this paper, we introduce MEG-GPT-2, a decoder-only transformer-based foundation model trained on source-level MEG data using a self-supervised autoregressive objective with factorised channel-time attention. Post-hoc analyses show that the model successfully captures temporal dynamics and spectral structure of brain activity while encoding anatomically meaningful spatial organisation and partially recovering amplitude-based functional connectivity. However, phase-sensitive inter-channel dependencies remain insufficiently modelled, highlighting limitations in capturing higher-order dynamics under short context lengths. Our results demonstrate that generative and representation-level evaluations provide complementary insights beyond downstream performance, offering a framework for assessing the neuroscientific validity of foundation models.