Predicting the Emergence of Induction Heads in Language Model Pretraining

Specialized attention heads dubbed induction heads (IHs) have been argued to underlie the remarkable in-context learning capabilities of modern language models; yet, a precise characterization of their emergence, especially in the context of language modeling, remains wanting. In this study, we investigate the relationship between statistical properties of the training data and IH formation in both natural and synthetic training data settings. We show that: (1) a simple equation combining batch size and context size predicts the point at which IHs form and that this emergence point is agnostic to model size; (2) surface bigram repetition frequency and reliability strongly affect the formation of IHs, and we find an effective decision boundary in terms of these two values; (3) local dependency with high bigram repetition frequency and reliability is sufficient for IH formation, but categoriality and the shape of the marginal distribution appear to modulate IH formation near the decision boundary.

Large language models can often learn from examples given in their input, but we still do not fully understand when this ability appears during training or what properties of the training data cause it to emerge. One proposed mechanism behind this ability is a special type of attention pattern called an induction head, which helps the model copy and continue repeated patterns in context. In this work, we study when induction heads form during language model pretraining and what kinds of data make them more likely to appear. Using both natural text and carefully controlled synthetic data, we find that a simple equation based on batch size and context size can predict when induction heads emerge. We also show that repeated word-pair patterns matter: induction heads are more likely to form when such repetitions are frequent and when the second word in the pair reliably follows the first. Finally, we find that other properties of the data, such as category structure and word-frequency distribution, can influence induction head formation near the boundary where emergence becomes uncertain. These findings help explain how language models acquire an important in-context learning mechanism and may inform more efficient choices of training data and training configuration.