Scalable and Interpretable Representation Alignment with Ordinal Similarity

Evaluating representation similarity is fundamental to representation learning. However, existing metrics suffer from significant limitations: they are difficult to interpret due to shifting baselines, lack robustness to outliers, and are frequently computationally intractable for large datasets, forcing a reliance on heuristic approximations. To address these shortcomings, we develop an ordinal-similarity framework, instantiated by the Triplet (TSI) and Quadruplet (QSI) Similarity Indices, which measure alignment by quantifying the consistency of ordinal relationships. We provide a theoretical analysis demonstrating that this formulation is inherently interpretable, robust to outliers, and computationally efficient. Finally, we establish a formal equivalence between TSI alignment and the alignment of local neighborhood structures, as measured by Mutual Nearest Neighbors. Through empirical analysis, we validate these properties and show that ordinal similarity offers a scalable, practical approach to measuring alignment, enabling practitioners to better understand and design representations.