Scale-Equivariant Alignment: Closing the Residual Barrier After Permutation Matching

Recent work shows that aligning permutation symmetries between two independently trained networks largely eliminates the loss barrier on linear interpolations (Ainsworth et al., 2023). A residual barrier persists, however, which Jordan et al. (2023) attribute to variance collapse — a phenomenon we show is a direct consequence of unresolved scale symmetries. ReLU networks possess a layerwise symmetry group that combines neuron permutations with positive rescalings, but existing alignment methods account only for permutations. We formally decompose the loss barrier into a permutation component and a scale component, prove a lower bound on the scale component in terms of mismatches in neuron scales between networks, and show that the optimal scale alignment is given by the geometric mean of per-neuron weight norms. Building on this, we propose Scale-Equivariant Alignment (SEA), an alternating algorithm that jointly solves for permutations and scales, and prove it converges monotonically to a critical point of a joint alignment objective. On CIFAR-10 ResNets, SEA reduces the residual loss barrier after permutation alignment by 74% and matches the accuracy of REPAIR without any test-time renormalization overhead.