Steering at the Source: Style Modulation Heads for Robust Persona Control

Activation steering offers a computationally efficient mechanism for controlling Large Language Models (LLMs) without fine-tuning. While effectively controlling target traits (e.g., persona), coherency degradation remains a major obstacle to safety and practical deployment. We hypothesize that this degradation stems from intervening on the residual stream, which indiscriminately affects aggregated features and inadvertently amplifies off-target noise. In this work, we identify a sparse subset of attention heads (only three heads) that independently govern persona and style formation, which we term Style Modulation Heads. Specifically, these heads can be localized via geometric analysis of internal representations, combining layer-wise cosine similarity and head-wise contribution scores. We demonstrate that intervention targeting only these specific heads achieves robust behavioral control while significantly mitigating the coherency degradation observed in residual stream steering. More broadly, our findings show that precise, component-level localization enables safer and more precise model control.

Controlling the personality and behavior of Large Language Models (LLMs) is crucial for making AI safer and more helpful, such as preventing harmful outputs or giving AI a specific conversational character. A popular method to achieve this involves injecting a "control signal" into the AI's internal thought process. However, this global intervention often severely disrupts the AI's ability to generate coherent and logical sentences, causing the text quality to collapse. In this work, we discovered that we do not need to alter the entire model. Instead, a highly specific and tiny subset of internal components (just three specialized "attention heads") independently governs the formation of personality and speaking style. By precisely targeting only these few components, we successfully control the AI's persona while perfectly preserving its ability to write clear, natural, and logical text. This finding moves us closer to practical and safe AI systems that can adapt their tone without losing their fundamental reasoning capabilities.