Scaling Vision Transformers for Functional MRI with Flat Maps

We study the problem of training self-supervised foundation models for functional MRI. Our main contributions are: (1) we introduce a new model family (CortexMAE) trained using the masked autoencoder framework on 2.1K hours of open fMRI data, and (2) we release the first open evaluation suite (Brainmarks) for fMRI foundation models. Our core innovation is simple: we adapt the Vision Transformer to fMRI by first converting each 3D fMRI volume to a 2D map using a cortical flat map projection. We directly compare flat maps to both parcellation and volume-based representations. While each has its advantages, flat maps generally perform best. We perform the first systematic scaling analysis for fMRI and observe strict power law scaling, albeit with limits. Finally, we use Brainmarks to do controlled benchmark comparisons. On subject-level trait prediction, we report a challenging null result: no single model achieves clear state-of-the-art performance. Moreover, all models struggle to outperform a simple functional connectivity baseline. On cognitive state decoding, we observe more robust performance, and in this setting our CortexMAE family outperforms prior models by a large margin. Code, models, and datasets are available at https://github.com/MedARC-AI/CortexMAE and https://github.com/MedARC-AI/Brainmarks.

Functional MRI (fMRI) lets us construct detailed 3D "videos" of a person's brain activity, completely noninvasively and using only magnets. Our goal is to build models that can extract clinically useful information from these remarkable, but also very challenging data. We propose a simple but effective idea: we transform the 3D fMRI "volumes" into a 2D images by flattening out all the brain's intricate folds (much like a globe is flattened into a 2D map). Once flattened, we can train models on fMRI just like we do for everyday images and videos. Our model achieves some promising results. For example, when we apply our model to fMRI recordings of people looking at pictures, we are able to "decode" what they're looking at (e.g. a picture of a dog, or a slice of pizza) more accurately than previous models. But there is still a lot of room for improvement. To help our field grow, we release all of our code, models, and datasets so that others can hopefully build on our work.