Grounding Functional Similarity by Invariance-Aware Model Stitching

In deep learning, functional similarity evaluation quantifies the extent to which independently trained models learn similar input--output relationships. In model stitching, functional similarity is framed as representation forward compatibility, i.e., whether the representations of two models can be aligned to solve a given task. Recent studies, however, highlight a critical limitation: models relying on different information cues can still produce compatible representations, making them appear misleadingly similar (Smith et al., 2025). We attribute this failure to standard model stitching being inherently blind to the invariance properties of the stitched models. To address this limitation, we introduce the forward--backward compatibility requirement under which we formulate the invariance-aware model stitching. Through analyzing key stitching configurations, we study the interplay between forward and backward compatibility, showing that invariance-aware model stitching provides a more principled approach to functional similarity evaluation while revealing functional discrepancies previously obscured.

Recent advancement in artificial intelligence can be largely attributed to deep learning models and their ability to learn to solve complex tasks when provided with large amounts of relevant data. To carry out their tasks, these models perform a series of complex computations, forming a strategy that ultimately transforms their inputs into task-relevant outputs. Given two models that have undergone independent learning processes, how can one quantify whether they have developed similar strategies for solving their, potentially different, tasks? Answering this is of great importance, as it enables a better understanding of the relationship between learning algorithms, data and the resulting strategies. Towards this goal, model stitching is a popular method where the internals of two models are connected to evaluate whether they can adequately perform the respective task. However, a recent study surfaced a key limitation of this approach where models known to employ different strategies appear to be misleadingly similar. To account for this undesirable behavior, we propose invariance-aware model stitching where we simultaneously evaluate whether the internals can be connected to solve the task and whether inputs found to produce similar internals with respect to one model do so for the other as well. Under this bidirectional perspective, we reveal discrepancies in models' strategies that were previously obscured.