BIT-LLM: Brain Instruction Tuned LLM with persistent Cross-Attention for fMRI-to-Text Decoding

Decoding fMRI into natural language is challenging because strong, pre-trained language priors can dominate autoregressive generation, obscuring whether a model truly utilizes neural evidence. We introduce BIT-LLM, which exposes fMRI-derived tokens as a persistent key–value memory through interleaved cross-attention adapters, enabling repeated neural access throughout decoding. BIT-LLM is trained with a three-stage pipeline: (i) multimodal contrastive learning to obtain semantically aligned fMRI representations, (ii) supervised fine-tuning to learn the brain-LLM interface while freezing the encoder and backbone LLM, and (iii) reward-based finetuning to optimize sequence-level caption quality directly. On the NSD subject-heldout benchmark (S1-7 train, S8 test), BIT-LLM yields substantially improved captioning quality over prior baselines under greedy decoding. In addition to standard captioning metrics, we perform several complementary evaluations to assess the robustness of brain–language grounding. Specifically, we conduct perturbation-based sanity checks by zeroing fMRI inputs or shuffling voxel values, and examine whether internal representations and generated outputs change accordingly. BIT-LLM exhibits clear sensitivity to these perturbations, indicating effective utilization of voxel values and their spatial correspondence.

When people look at a picture, their brain activity forms a pattern that can be measured with MRI scanners. A long-standing goal is to use those patterns to generate a plain-language description of what the person saw. This is difficult because modern language models can produce fluent descriptions even when they are not truly using the brain signal. BIT-LLM addresses this problem by giving the language model repeated access to brain-derived information while it writes each caption, rather than using the brain signal only as a short prompt at the beginning. We tested the method on a public brain-imaging dataset: the model was trained on data from seven people and evaluated on an eighth person it had never seen during training. BIT-LLM generated more accurate descriptions than previous methods for turning brain activity into text. We also tested whether the model was relying on brain activity by removing or scrambling the brain signal at inference time. In those tests, performance dropped substantially, suggesting that the generated captions depend on meaningful neural information rather than language patterns alone.