Geometry of Ordinal Representations in Language Models

Recent work showed that language models represent character counts on curved 1D manifolds, with attention heads performing geometric transformations to enable computation. We test whether this generalizes across four ordinal tasks (bracket depth, indentation, table position, numeric magnitude) in Gemma-2-2B, Gemma-2-9B, and Qwen3-4B. We find that 1D manifolds with place-cell feature tiling emerge for tasks where the ordinal variable is locally computable from token identity, while tasks requiring cross-position integration or semantic extraction produce higher-dimensional or incoherent representations. Geometric computation is architecture-dependent: Qwen3-4B shows 4--10$\times$ stronger twisting than Gemma models for indentation, and its twisters preserve ordinal order ($\rho \geq 0.77$), unlike its numeric twisters ($\rho \leq 0.23$). Activation patching causally confirms that the manifold subspaces carry task-relevant information across all settings.