Parameter symmetries determine representational geometry in overparameterized nonlinear networks

Representations are routinely used in machine learning, psychology, and neuroscience to probe the computations of biological and artificial systems. Yet it remains unclear to what extent computation constrains representation in artificial neural networks. Neural networks admit parameter symmetries: transformations of the parameters that preserve function exactly but reshape representational geometry. We show that known parameter symmetries act on representations through just three primitives: addition, duplication, and scaling, giving a closed-form descriptor of representational geometry as task-linked features plus symmetry-induced noise. This decomposition yields analytic bounds on representational similarity between networks related by parameter symmetry, which reveal when functionally equivalent networks become arbitrarily dissimilar. Finally, we identify privileged representational geometries, which weight features by their computational importance and recover a stable link between representation and computation. Overall, our results delineate when representation can support inferences about computation, and when it cannot.