Scalable and Interpretable Representation Alignment with Ordinal Similarity

Evaluating representation similarity is fundamental to representation learning. However, existing metrics suffer from significant limitations: they lack interpretability due to shifting baselines, lack robustness to outliers, and are computationally intractable for large datasets, forcing reliance on heuristic approximations. To address this, we develop an ordinal-similarity framework, instantiated by the Triplet (TSI) and Quadruplet (QSI) Similarity Indices, which measure alignment by quantifying the consistency of ordinal relationships. We theoretically demonstrate this formulation is inherently interpretable, robust to outliers, and computationally efficient. Finally, we establish a formal equivalence between TSI and local neighborhood alignment, measured by Mutual Nearest Neighbors. Empirically, we validate these properties and show that ordinal similarity offers a scalable approach to measuring alignment, enabling practitioners to better understand and design representations.

As AI systems become increasingly complex, it is crucial to understand if different models interpret data in similar ways. However, current tools for measuring how closely their internal representations align are difficult to interpret, easily thrown off by unusual data points, and too slow to process large, modern datasets. To solve this, we developed a new approach called "ordinal similarity" that measures alignment based on relative comparisons rather than exact numerical values. Much like asking if a dog is more similar to a wolf than to a cat, our method—using the Triplet and Quadruplet Similarity Indices—checks if the relative ranking of similarities between items remains consistent across different AI models. We mathematically prove that these new indices are naturally easy to interpret, highly resistant to extreme outliers, and exceptionally fast to compute. Ultimately, this provides researchers with a reliable, scalable way to look inside the "black box" of AI, helping them design more transparent systems that genuinely align with human reasoning.