Connecting Independently Trained Modes via Layer-Wise Connectivity

Empirical studies have shown that continuous low-loss paths can be constructed between independently trained neural network models. This phenomenon, known as mode connectivity, refers to the existence of such paths between distinct modes-i.e., well-trained solutions in parameter space. However, existing empirical methods do not reliably connect independently trained modes and have been evaluated mainly on a narrow set of architectures (e.g., basic CNNs, VGG, and ResNet), leaving their effectiveness on newer models unclear. In this work, we propose a new empirical algorithm for connecting independently trained modes that generalizes beyond traditional architectures and supports a broader range of networks, including MobileNet, ShuffleNet, EfficientNet, RegNet, Deep Layer Aggregation (DLA), and Compact Convolutional Transformers (CCT). In addition to broader applicability, the proposed method yields more consistent connectivity paths across independently trained mode pairs and supports connecting modes obtained with different training hyperparameters.

Independently trained neural networks are usually considered uncorrelated and located at isolated minima in the loss landscape. However, prior work in 2018 showed that these models can often be connected through a continuous low-loss path, although understanding why and under what conditions such paths exist remains an open problem. In this work, we propose a new method for connecting independently trained models that provides more stable reproducibility, lower maximum loss along the connection path, and support for a broader range of modern architectures, including transformer-based models. Our results further suggest that independently trained models belong to a larger fully-path-connected low-loss region rather than isolated solutions.