Partial Fusion of Neural Networks: Efficient Tradeoffs Between Ensembles and Weight Aggregation

Ensembles of neural networks typically outperform individual networks but incur large computational costs, whereas weight aggregation produces less costly, yet also less accurate, aggregate models. We introduce partial fusion of networks, which interpolates between ensembles and weight aggregation and thus allows for a flexible tradeoff between computational cost and performance. A direct way to achieve this is to extend existing weight aggregation methods based on neuron-level similarity between different networks, where partial fusion then only aggregates weights of neurons which are most similar. We showcase one particular method to jointly identify which neurons are most similar and match them via partial optimal transport. Further, we consider the more general perspective of weight aggregation and partial fusion as generalized pruning of ensemble models, where neurons cannot just be deleted, but also linearly combined. Finally, we show that generalized pruning applied to a single network yields similar benefits as partial fusion by allowing for a tradeoff between isolating, deleting, and linearly combining neurons based on similarity.

When multiple AI models based on neural networks are combined, there are two common approaches: running them all separately and averaging their answers, which is accurate but expensive, or blending their internal parameters into a single model, which is cheap but often less accurate. Until now, there was no principled way to get something in between. We introduce partial fusion, which merges only the parts of two models that are doing similar work internally, while keeping the unique parts separate. To find which parts are similar, we use a mathematical matching technique from optimal transport theory. We also show that the same idea works for making a single model smaller: instead of simply removing less important components, we can merge similar ones together, preserving more of what the model has learned. In our experiments, partially fused models use fewer resources than running two full models, yet perform nearly as well. This provides a flexible way to balance computational cost against performance, which is especially valuable when deploying AI systems with limited resources. More broadly, our work shows that it is meaningful to measure whether two neural networks are partly similar in their reasoning, which yields insights into the internal mechanisms of neural networks.