Fine-grained Analysis of Brain-LLM Alignment through Input Attribution

Understanding the alignment between large language models (LLMs) and human brain activity can reveal computational principles underlying language processing. This work describes a pipeline to apply attribution methods to the brain-LLM alignment setting to identify the specific words most important for this alignment. As a case study, we leverage it to study a contentious research question about brain-LLM alignment: the relationship between brain alignment (BA) and next-word prediction (NWP). Across two naturalistic fMRI datasets, we find that BA and NWP rely on largely distinct word subsets: NWP exhibits recency and primacy biases with a focus on syntax, while BA prioritizes semantic and discourse-level information with a more targeted recency effect. This work advances our understanding of how LLMs relate to human language processing and highlights differences in feature reliance between BA and NWP. Beyond this study, our attribution method can be broadly applied to explore the cognitive relevance of model predictions in diverse language processing tasks.

Large language models (LLMs) can predict patterns of brain activity recorded while people read the same text, a phenomenon known as brain alignment. Why does this alignment occur? We developed a method to identify which words models rely on when predicting brain activity. While this framework can be broadly used to study brain alignment, we used it to investigate whether it mainly depends on the models’ ability to predict the next word. We found that the two tasks rely on very different kinds of information drawn from different parts of the text. Predicting the next word depends strongly on syntax and on words appearing at the very beginning and end of the context. Predicting brain activity, instead, relies more on meaning and story-level information, with a broader focus on words near the end of the context. These findings suggest that brain alignment cannot be explained by next-word prediction alone and instead reflects richer language processing.