Model Fusion via Retrofitting

Model fusion seeks to combine independently trained neural networks into a single model without retraining, but is complicated by representational divergence arising from permutation invariance, random initialization, and heterogeneous training data. Existing methods struggle particularly in zero-shot settings under non-IID data distributions, and are often limited to specific architectures or pairwise fusion. We introduce a neuron-centric family of fusion algorithms that frames fusion as a principled representation-matching problem: intermediate neurons across parent models are grouped into target representations, which the fused model's corresponding sub-networks are then trained to approximate. Unlike prior work, our approach incorporates neuron attribution scores to bias alignment toward salient features, and can be applied to any architecture modularizable as a DAG of levels—empirically validated on VGGs, ResNets, and ViTs. Experiments across standard benchmarks show consistent improvements over existing fusion methods, with the largest gains in zero-shot and non-IID scenarios. Code is available at https://github.com/AndrewSpano/model-fusion-via-retrofitting}{https://github.com/AndrewSpano/model-fusion-via-retrofitting.

When we have more than one deep neural network, how should we use all of them? A simple approach is to ensemble the parent models, but then inference cost scales with the number of models. Model fusion instead aims to combine them into a single model. We developed a family of fusion algorithms that represent groups of neurons from the parent models with neurons in the fused model. Our approach processes the base models by user-defined levels. For each level, we group parent neurons together to make targets, either by pairing them up one to one or by clustering similar ones, and finally fit the fused model's neurons to approximate the targets. We also incorporate neuron importance scores, which reflect how much each neuron contributes to the model's predictions, to improve fusion quality. We tested our method on VGGs, ResNets, and ViTs and found that it outperforms existing approaches, especially when no further training is possible or when the training data cannot be shared, for example, due to privacy constraints.