Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability

Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged. Despite growing attention to structural parameter symmetries, the exact interplay between parameters, data, and representations remains underexplored. To investigate this, we develop a theoretical framework of effective function classes defined by the neurons' induced functions restricted to the representation subspace. We then formalize effective symmetry breaking via neuron identifiability across independent training runs. Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in structurally asymmetric models. This allows us to disentangle the effects of data-specific and architectural symmetries. We further show that neuron identifiability enables representation merging without prior alignment, and characterize when such merging admits a linear low-loss connecting path. These findings highlight the role of effective function classes in affecting the loss landscape.