Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability

Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged. Despite growing attention to parameter symmetries, the exact interplay between parameters, data, and representations remains underexplored. To investigate this, we develop a theoretical framework of effective function classes, i.e., the set of functions a neuron can realize on its input support, and the norm cost of realizing them. We then formalize effective symmetry breaking via neuron identifiability across independent training runs. Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in structurally asymmetric models. We further show that neuron identifiability enables representation merging without prior alignment, and characterize when such merging admits a linear low-loss path. These findings highlight the role of effective function classes in affecting the loss landscape.

Many machine learning models have symmetries in their parameters: This means that there can be multiple models which behave exactly the same, but look very different in their internals. The parameters of a model are a long list (thousands or millions) of numbers that make up the model. A symmetry in parameter space means that several different parameter lists all yield models that behaves exactly the same, even though the parameter lists look very different. Machine learning researchers have hypothesized that different trained models are essentially very similar, but these symmetries make their parameters look different. Previously, researches have successfully tried to break such symmetries, such that when you train two models, they will always end up to not only behave similarly, but also have similar parameters. We now find examples where this fails, and analyze why: Two models may behave very similarly on the data they usually process, while technically they are dissimilar because they behave differently on more obscure inputs. Because of that, even with symmetry breaking, there can still be models that behave the same (on the data we care about), but seem different in parameter space.