Translation Heads: Disentangling meaning from language in LLM-based machine translation

Mechanistic Interpretability (MI) seeks to explain how neural networks implement their capabilities, but the scale of Large Language Models (LLMs) has limited prior MI work in Machine Translation (MT) to word-level analyses. We study sentence-level MT from a mechanistic perspective by analyzing attention heads to understand how LLMs internally encode and distribute translation functions. We decompose MT into two subtasks: producing text in the target language (i.e. target language identification) and preserving the input sentence’s meaning (i.e. sentence equivalence). Across three families of open-source models and 20 translation directions, we find that distinct, sparse sets of attention heads specialize in each subtask. Based on this insight, we construct subtask-specific steering vectors and show that modifying just 1% of the relevant heads enables instruction-free MT performance comparable to instruction-based prompting, while ablating these heads selectively disrupts their corresponding translation functions.

When AI systems like chatGPT translate between languages, we don't fully understand what happens inside them. This makes it hard to diagnose errors or improve their performance. We investigated how large language models organize the task of translation internally, by looking at their components called "attention heads" that help the model decide what to focus on. We discovered that translation can be split into two distinct jobs: picking the right output language and preserving the meaning of the original sentence. The model assigns each job to a separate, small group of components. By activating just these few components, we can make a model translate without ever giving it explicit instructions. Conversely, disabling them breaks translation in predictable ways: removing the language-picking components causes the model to mix languages mid-sentence, while removing the meaning components produces fluent but inaccurate output. These findings, confirmed across three model families and twenty language pairs, improve our understanding of how AI systems handle translation and open new paths toward more controllable and interpretable systems.