How Do Language Models Speak Languages? A Case Study on Unintended Code-Switching

Unintended code-switching, where LLMs unexpectedly switch languages, poses a fundamental challenge to multilingual generation in LLMs. However, we still lack a mechanistic account of how this failure mode is implemented inside the model. Key questions remain: what internal components (i.e., circuits) give rise to unintended code-switching, where do they emerge across layers, and how can we intervene to mitigate it? In this work, we introduce a scalable circuit discovery framework that causally localizes multilingual neurons and describes their functional patterns, then further groups them into interpretable circuits—without any additional training or manual annotation. Our findings are twofold: a) The model's "speaking-a-language" circuit decomposes into a language regime (detecting and maintaining language identity) and a semantic regime (retrieving language-agnostic semantics). b) The mechanism of unintended code-switching is a regime shift. The semantic regime suppresses the language regime and overwhelms the multilingual circuit, causing the model to generate in an unintended language. To validate these findings, we further fine-tune the identified language sub-circuit, reducing the code-switching rate by $20.8\%$ with minimal parameter updates ($\sim0.019$ % of all neurons). This work serves as a preliminary exploration of multilingual generation mechanism, offering actionable insight for targeted training for multilingual LLMs.

Large language models can answer questions and write text in many languages, but they sometimes unexpectedly switch to another language in the middle of a response. This behavior, called unintended code-switching, can make multilingual systems less reliable and harder to trust, especially for users who expect answers in a specific language. We study why this happens inside language models. Instead of treating the model as a black box, we identify small groups of internal neurons that help the model keep track of the intended language. We find that multilingual generation depends on two interacting processes: one that maintains the target language, and another that represents the meaning of the response across languages. Code-switching tends to occur when the meaning-related process becomes too influential and weakens language-specific control. This understanding also suggests a practical fix. By fine-tuning only a very small set of language-related neurons, we can reduce unintended code-switching while leaving most of the model unchanged. Our work helps explain a common failure in multilingual language models and points toward more targeted ways to make them safer and more consistent for users across languages.