A Versatile Influence Function for Data Attribution with Non-Decomposable Loss

Influence function, a technique rooted in robust statistics, has been adapted in modern machine learning for a novel application: data attribution---quantifying how individual training data points affect a model's predictions. However, the common derivation of influence functions in the data attribution literature is limited to loss functions that decompose into a sum of individual data point losses, with the most prominent examples known as M-estimators. This restricts the application of influence functions to more complex learning objectives, which we refer to as non-decomposable losses, such as contrastive or ranking losses, where a unit loss term depends on multiple data points and cannot be decomposed further. In this work, we bridge this gap by revisiting the general formulation of influence function from robust statistics, which extends beyond M-estimators. Based on this formulation, we propose a novel method, the Versatile Influence Function (VIF), that can be straightforwardly applied to machine learning models trained with any non-decomposable loss. In comparison to the classical approach in statistics, the proposed VIF is designed to fully leverage the power of auto-differentiation, hereby eliminating the need for case-specific derivations of each loss function. We demonstrate the effectiveness of VIF across three examples: Cox regression for survival analysis, node embedding for network analysis, and listwise learning-to-rank for information retrieval. In all cases, the influence estimated by VIF closely resembles the results obtained by brute-force leave-one-out retraining, while being up to 1000 times faster to compute. We believe VIF represents a significant advancement in data attribution, enabling efficient influence-function-based attribution across a wide range of machine learning paradigms, with broad potential for practical use cases.

We want to understand how each individual training data point influences a machine learning model's behavior. One existing and popular tool for this task, called influence function, works only with setups where the model's training loss can be split into sum of units where each unit only depends on a single training data point. This leaves out many machine learning tasks, like ranking retrieval results or learning network structures, whose loss depends on interactions between training data points.To fix this, we explore and develop a new method, called the Versatile Influence Function (VIF), that works even when training losses can not be split into single units. This method could easily leverage automatic differentiation, the same tool that powers machine learning model training, to avoid case-specific derivations of each loss function.We show that VIF works well in several real-world tasks like predicting survival time, network embedding, and ranking retrieval results. Our work makes it easier for the community to understand and trust machine learning models.