Interpretable Self-Supervised Learning via Representer Landmarks and Nyström Approximation

Self-supervised learning (SSL) learns representations from massive unlabeled data, yet the resulting models typically operate as black boxes, necessitating domain-specific explanations. We introduce KREPES, a unified framework to analytically interpret the learned representations of SSL objectives, including SimCLR, BYOL, and VICReg. By bridging empirical neural tangent kernel approximations of neural networks with the Representer Theorem for kernels, we express the learned latent space directly via "Representer Landmarks", which are the representations of influential unlabeled training examples. We introduce novel metrics, "Sample-Specific Influence Score", "Concept-Conditioned Influence Score" and "Feature Alignment Gap", to quantify the transparency of the learned representations. KREPES enables direct audit of the latent space without supervision, for example, revealing an algorithmic bias in the Adult-1M dataset where SSL uses demographic proxies for income. Finally, to ensure scalability to benchmarks with 1M+ samples (ImageNet-1K, Adult-1M), KREPES introduces a novel Nyström approximation-based analytical inference framework for SSL objectives.

Machine learning models can learn from massive amounts of data, without human-provided labels in a process called self-supervised learning. However, these powerful models operate as "black boxes”, which means we know they work, but we don’t understand why they organize and group data the way they do. We introduce KREPES, a framework that makes these black boxes transparent. To explain how an AI views a new piece of data, KREPES mathematically traces the AI's internal logic back to specific, highly influential training examples called "Representer Landmarks". For instance, if the AI groups a new image into a specific cluster, our metrics can reveal the exact training examples, as well as the human understandable concepts on which the model relied that anchored that decision. Crucially, analyzing models trained on millions of examples is usually too computationally expensive. We introduce a mathematical shortcut to make this scalable to large datasets. By revealing the hidden structure of the AI's learned space, KREPES allows researchers to directly audit models for algorithmic biases, such as relying on demographic proxies, even when no labels were used.