The Cylindrical Representation Hypothesis for Language Model Steering

Steering is widely used for controlling large language models, yet its effects are often unstable and difficult to predict. Existing theoretical accounts are largely based on the Linear Representation Hypothesis (LRH), which assumes that concepts can be orthogonalized for lossless control. However, this assumption rarely holds in practice and cannot explain the variability of steering outcomes. We propose the Cylindrical Representation Hypothesis (CRH), a geometric extension of LRH that relaxes the orthogonality assumption while preserving linear concept representations. We show that overlapping concept contributions naturally induce a sample-specific cylindrical structure consisting of a central axis, a normal plane, and sensitive sectors. The central axis captures the primary semantic transition associated with a target concept, while the normal plane governs steering sensitivity. Within this plane, some sectors facilitate concept activation, while others suppress or delay it. CRH reveals an asymmetry in steering predictability: the normal plane can be inferred from difference vectors, but the sensitive sectors cannot, introducing an intrinsic source of uncertainty. This explains why steering outcomes vary across samples even when intervention directions are well aligned. Experiments spanning 100 concepts, multiple models, and diverse steering methods provide consistent evidence for the predicted cylindrical structure, suggesting that steering variability arises from representation geometry rather than imperfect steering vectors. Our code is available at: https://github.com/mbzuai-nlp/CRH.

Steering is a simple way to guide a large language model by adding a control vector inside the model. It can make the model produce text with a target style, topic, or behavior. However, steering is often unstable. The same method may work for one input but fail for another. This work studies why this happens. Existing explanations often rely on the Linear Representation Hypothesis. This view assumes that each concept is represented as a clean direction in the model. It also assumes that different concepts can be separated into independent directions. In real models, this assumption is too ideal. Many concepts overlap with each other, so steering can be affected by hidden interference. We propose the Cylindrical Representation Hypothesis as a more realistic explanation. For each input, steering forms a local cylindrical structure. The central axis gives the main direction toward the target concept. The normal plane captures interference from other concepts. Some regions in this plane make steering easier, while others delay or suppress the target concept. This explains why two similar steering vectors can lead to very different outputs. We test this idea on 100 concepts, multiple model architectures, and several steering methods. The results support the proposed structure and suggest a new way to understand and improve controllable language models.