From Directions to Regions: Decomposing Activations in Language Models via Local Geometry

Activation decomposition methods in language models are tightly coupled to geometric assumptions on how concepts are realized in activation space. Existing approaches search for individual global directions, implicitly assuming linear separability, which overlooks concepts with nonlinear or multi-dimensional structure. In this work, we leverage Mixture of Factor Analyzers (MFA) as a scalable, unsupervised alternative that models the activation space as a collection of Gaussian regions with their local covariance structure. MFA decomposes activations into two compositional geometric objects: the region's centroid in activation space, and the local variation from the centroid. We train large-scale MFAs for Llama-3.1-8B and Gemma-2-2B, and show they capture complex, nonlinear structures in activation space. Moreover, evaluations on localization and steering benchmarks show that MFA outperforms unsupervised baselines, is competitive with supervised localization methods, and often achieves stronger steering performance than sparse autoencoders. Together, our findings position local geometry, expressed through subspaces, as a promising unit of analysis for scalable concept discovery and model control, accounting for complex structures that isolated directions fail to capture.

Many methods for interpreting language models search for directions in the model’s hidden space that correspond to concepts. However, many concepts have richer, nonlinear structure that these methods can miss. We introduce an unsupervised local-geometry view of activation space, modeling representations as a collection of regions, each with its own low-dimensional structure. This decomposes activations into two interpretable components: where they are located in the model’s internal space and how they can vary around that location. Across Llama-3.1-8B and Gemma-2-2B, this view reveals meaningful concept-relevant regions and improves localization and steering over strong unsupervised baselines, while remaining competitive with supervised methods in several settings. These results suggest that local geometry is a practical unit for interpreting and controlling language models beyond isolated global directions.