Functional Alignment Can Mislead: Examining Model Stitching

A common belief in the representational comparison literature is that if two representations can be functionally aligned, they must capture similar information. In this paper we focus on model stitching and show that models can be functionally aligned, but represent very different information. Firstly, we show that discriminative models with very different biases can be stitched together. We then show that models trained to solve entirely different tasks on different data modalities, and even clustered random noise, can be successfully stitched into MNIST or ImageNet-trained models. We end with a discussion of the wider impact of our results on the community's current beliefs. Overall, our paper draws attention to the need to correctly interpret the results of such functional similarity measures and highlights the need for approaches that capture informational similarity.

With the widespread adoption of AI, a better understanding of neural networks is becoming a priority. Researchers can measure how well neural networks perform a task, but would also like to understand how different networks perform the task. A useful tool that might help shed light on the inner workings of neural networks would be one that meaningfully determines when two networks are different. In this work, we focus on a technique called stitching. Informally, stitching strongly focuses on what decision a network makes. We argue that this focus can abstract away from how the network makes that decision. As a result, stitching can consider two networks to be similar if they make the same decision but based on very different evidence (or rules). This becomes problematic since not all rules are equally likely to hold when networks are deployed in the wild. Thus, networks can have qualitatively different performance in the real world but can falsely be considered similar if compared using approaches such as stitching. In our paper we create a series of carefully designed experiments to illustrate this and encourage the deep learning community to consider such settings when evaluating other network comparison tools.