Concept Concentration for Faithful Representation Intervention

Representation intervention aims to locate and modify the representations that encode the underlying concepts in Large Language Models (LLMs) to elicit the aligned and expected behaviors. Despite the empirical success, it has never been examined whether one could locate the faithful concepts for intervention. In this work, we explore the question in safety alignment. If the interventions are faithful, the intervened LLMs should erase the harmful concepts and be robust to both in-distribution adversarial prompts and the \textit{out-of-distribution} (OOD) jailbreaks. While it is feasible to erase harmful concepts without degrading the benign utility of LLMs in linear settings, we show that it is \textit{infeasible} in the general non-linear setting. To tackle the issue, we propose \texttt{Concept Concentration} (\texttt{COCA}). \texttt{COCA} refactors the training data with an explicit reasoning process, which first identifies the potential unsafe concepts and then decides the responses. Essentially, \texttt{COCA} simplifies the decision boundary between harmful and benign representations, enabling more effective linear erasure. Extensive experiments with multiple representation intervention methods and model architectures demonstrate that \texttt{COCA} significantly reduces both in-distribution and OOD jailbreak success rates, and meanwhile maintaining strong performance on regular tasks such as math and code generation. Our code is publicly available at: \url{https://github.com/tmlr-group/COCA}.

While large language models (LLMs) are trained to refuse harmful requests, they remain vulnerable to jailbreak prompts that obscure unsafe intents. This paper investigates why representation intervention, a technique that directly intervenes in how a model internally encodes concepts, often fails to ensure safety alignment. The core issue is that a model's internal representations of harmful and benign concepts can become entangled, making it difficult to erase unsafe behaviors without degrading general utility. To address this, we propose COCA (Concept Concentration). COCA trains the model using a structured reasoning process. It first identifies the underlying concept in a request, evaluates it for harm, and then formulates a response. This explicit reasoning untangles and isolates unsafe concepts, allowing standard intervention methods to effectively block harmful prompts while preserving the model's helpful capabilities.