Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoders

Multilingual language models can solve the same mathematical reasoning problem in different languages, but it remains unclear whether they rely on shared internal features or on language-specific computations that only produce similar outputs. We study this question in five models from four architecture families using the Multilingual Grade School Math (MGSM) dataset, with problems solved independently in English, German, French, Spanish, Russian, and Chinese, retaining only problems with valid reasoning traces in all six languages and replaying those traces through the model to record internal representations at multiple layers. For each model, we first use Centered Kernel Alignment (CKA) to identify layers with strong cross-language alignment. At each selected layer, we train two sparse autoencoders: a baseline reconstruction-only model and a contrastive variant introduced in this work, the Geometry-Invariant Sparse Autoencoder (GI-SAE). GI-SAE supplements the reconstruction loss with an Information Noise-Contrastive Estimation (InfoNCE) loss that trains the encoder to produce similar feature activations for traces of the same problem, regardless of language or token position. We then test whether the resulting shared features are functionally interchangeable by swapping shared feature values between languages during the model's forward pass and measuring the resulting change in output (causal patching), quantified by Kullback-Leibler (KL) divergence per shared feature. Although GI-SAE yields higher CKA and Jaccard similarity at nearly every layer, higher geometric similarity does not consistently imply greater functional interchangeability across languages. We find that cross-language feature sharing is strongly model- and architecture-dependent in this sample and appears at different depths in different models. GI-SAE primarily amplifies cross-language structure already present in each model: the pattern is model-specific, with progressive strengthening in Qwen, no functional benefit in already-saturated Gemma, and mixed layer-dependent effects in Llama and Phi.