The Mechanistic Emergence of Symbol Grounding in Language Models

Symbol grounding (Harnad, 1990) describes how symbols such as words acquire their meanings by connecting to real-world sensorimotor experiences. Recent work has shown preliminary evidence that grounding may emerge in (vision-)language models trained at scale without using explicit grounding objectives. Yet, the specific loci of this emergence and the mechanisms that drive it remain largely unexplored. To address this problem, we introduce a controlled evaluation framework that systematically traces how symbol grounding arises within the internal computations through mechanistic and causal analysis. Our findings show that grounding concentrates in middle-layer computations and is implemented through the aggregate mechanism, where attention heads aggregate the environmental ground to support the prediction of linguistic forms. This phenomenon replicates in multimodal dialogue and across architectures (Transformers and state-space models), but not in unidirectional LSTMs. Our results provide behavioral and mechanistic evidence that symbol grounding can emerge in language models, with practical implications for predicting and potentially controlling the reliability of generation.

When people use a word like “horse”, its meaning is tied to things they have seen or experienced in the world. This connection between words and evidence from the world is referred to as grounding. In this work, we investigate if language models can do the same: we ask whether it is possible for grounding to emerge on its own during ordinary training, and if so, how. We study this question by creating controlled situations where the same idea appears in two separate forms: one as part of the environment and one as ordinary language. For example, a model may see evidence about a horse and then be asked to predict the word “horse” in a sentence. We compare cases where the evidence matches the word with cases where it does not, and we examine the model to identify which parts facilitate the correct prediction. We find that most modern model architectures, including transformers, learn this connection during ordinary training, even without being explicitly told which words match which evidence. The connection is strongest in the middle of the model and is carried by attention patterns that bring relevant environmental information to the word being predicted. We observe similar behavior in text-based and image-based dialogue settings, though not in a simpler recurrent model. These findings help explain how grounding can arise in language models and may offer tools for detecting or reducing unreliable language model generations.