Bridging Functional and Representational Similarity via Usable Information

We present a unified framework for quantifying the similarity between representations through the lens of \textit{usable} information, offering a rigorous theoretical and empirical synthesis across three key dimensions. First, addressing functional similarity, we establish a formal link between stitching performance and conditional mutual information. We further reveal that stitching is inherently asymmetric, demonstrating that robust functional comparison necessitates a bidirectional analysis rather than a unidirectional mapping. Second, concerning representational similarity, we find that reconstruction-based metrics and standard tools (e.g., CKA, RSA) act as estimators of usable information under specific constraints. Crucially, we show that similarity is relative to the capacity of the predictive family: representations that appear distinct to a rigid observer may be identical to a more expressive one. Third, we demonstrate that representational similarity is sufficient but not necessary for functional similarity. We unify these concepts through a task-granularity hierarchy: similarity on a complex task guarantees similarity on any coarser derivative, establishing representational similarity as the limit of maximum granularity: input reconstruction.

Deep learning models power modern AI by compressing complex data into internal summaries called "representations". Understanding whether two different models have learned the exact same representations is a major open question in both AI and cognitive science. Historically, researchers have answered this using two disconnected methods: checking if the models behave identically on a specific task (functional similarity), or checking if their internal data structures look mathematically identical (representational similarity). This paper provides a unified framework that bridges these two distinct perspectives using the concept of "usable information". Specifically, our research reveals three major insights: - First, we prove that swapping parts between models to test their compatibility is essentially a one-way street. A true functional comparison requires testing the swap in both directions. - Second, we show that popular geometric measurement tools are not arbitrary. Instead, they actually estimate how much information a specific model can successfully extract and use. - Third, we demonstrate a clear hierarchy: if two models share identical internal structures, they are guaranteed to behave the same on any derived task. However, the reverse is not true, because models can achieve the exact same functional outcome by ignoring different irrelevant details.