Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Models

Protein sequences are abundant in repeating segments, both as exact copies and as approximate segments with mutations. These repeats are important for protein structure and function, motivating decades of algorithmic work on repeat identification. Recent work has shown that protein language models (PLMs) identify repeats, by examining their behavior in masked-token prediction. To elucidate their internal mechanisms, we investigate how PLMs detect both exact and approximate repeats. We find that the mechanism for approximate repeats functionally subsumes that of exact repeats. We then characterize this mechanism, revealing two main stages: PLMs first build feature representations using both general positional attention heads and biologically specialized components, such as neurons that encode amino-acid similarity. Then, induction heads attend to aligned tokens across repeated segments, promoting the correct answer. Our results reveal how PLMs solve this biological task by combining language-based pattern matching with specialized biological knowledge, thereby establishing a basis for studying more complex evolutionary processes in PLMs.

Proteins often contain internal repeated segments that are important for their structure and function. Over evolution, these repeats accumulate mutations, so segments that may have started as identical often become only approximate matches. This makes repeat identification a long-standing challenge in computational biology. More recently, protein language models (PLMs) have been shown to detect such repeats and exploit them in masked-token prediction. But what internal mechanism enables this ability? In this work, we uncover how PLMs detect exact and approximate repeats. We find that PLMs rely on two categories of internal components. The first includes language-related components previously linked to repeat identification in text language models, such as attention heads that attend to matching positions across repeated segments. The second includes biologically specialized components, such as neurons that respond to biochemically similar amino-acid groups. We further reveal a three-stage mechanism showing how PLMs combine these components to solve the task. By revealing the internal mechanism behind repeat detection, our work provides a foundation for studying more complex evolutionary processes in PLMs. Understanding this mechanism may also help researchers build better repeat-detection tools and address failure modes such as repetitive generation in PLMs.