MEG-XL: Data-Efficient Brain-to-Text via Long-Context Pre-Training

Clinical brain-to-text interfaces are designed for paralysed patients who cannot provide extensive training recordings. Pre-training improves data-efficient generalisation by learning statistical priors across subjects, but these priors critically depend on context. While natural speech might unfold gradually over minutes, most methods pre-train with only a few seconds of context. Thus, we propose MEG-XL, a model pre-trained with 2.5 minutes of MEG context per sample, 5-300× longer than prior work, and equivalent to 191k tokens, capturing extended neural context. Fine-tuning on the task of word decoding from brain data, MEG-XL matches supervised performance with a fraction of the data (e.g. 1hr vs 50hrs) and outperforms brain foundation models. We find that models pre-trained with longer contexts learn representations that transfer better to word decoding. Our results indicate that long-context pre-training helps exploit extended neural context that other methods unnecessarily discard.

People who have lost the ability to speak—for instance, due to paralysis—could communicate again if a computer could read words from their brain activity. But these patients cannot easily provide the dozens of hours of training recordings current machine learning systems require. Training the model on lots of brain data from healthy volunteers should help, but existing attempts look at only a few seconds of brain activity at a time, losing access to helpful context. As an analogy, knowing the preceding words "the cat sat on the" makes it easier to predict that "mat" follows; the same is true of brain activity. We built MEG-XL, a model that learns from 2.5 minutes of brain activity per sample, 5 to 300 times longer than prior methods. It is trained by learning to reconstruct hidden chunks of the signal from the surrounding context. MEG-XL decodes words with roughly one hour of recordings whereas existing approaches may need up to fifty. We also find that simply showing long recording samples is not enough and a model has to be trained on long samples to learn what to ignore and what to attend to. This brings brain-to-text systems for speech restoration one step closer to clinical reality.