Barriers to Counterfactual Credit Attribution for Autoregressive Models

Generative AI disrupts the practice of giving credit to work that came before. Ideally, a generative model would give credit to any work on which its output depends in a significant way. Counterfactual credit attribution (CCA) is a technical condition formalizing this goal---a relaxation of differential privacy---recently introduced by Livni, Moran, Nissim, and Pabbaraju (2024) who studied it in the PAC learning setting. We initiate the study of CCA generative models. Specifically, we consider autoregressive models giving credit to a deployment-time dataset (e.g., a RAG database). We uncover barriers to two natural approaches to CCA autoregressive models. First, we show that imposing CCA on the underlying next-token predictor does not guarantee that the model is CCA: CCA does not compose autoregressively (unlike DP). Second, we consider a different approach to building CCA models which we call retrofitting. Retrofitting takes a model that does not attribute credit, and adds credit onto it. Given black-box access to the starting model, retrofitting requires query complexity exponential in the length of the model's outputs.

Generative AI systems produce output based on large amounts of existing work, but they often do not clearly acknowledge which sources contribute to the output. This paper studies the problem of ensuring generative models give proper credit to data. In particular, it focuses on the type of model that produces a document by predicting the next word one by one. Most large language models work roughly in this way. This paper considers a criterion for "what counts as giving proper credit" proposed by previous research inspired by data privacy. Under this criterion, a model gives proper credit if, whenever it does not give credit to a data point, that data point's contribution is negligible. The paper provides theoretical results that show that requiring models to give proper credit under this criterion is difficult: First, we show that even if the model gives proper credit when predicting the next word at each step, the overall text-generation process may still fail to do so. We also show that equipping a model to give proper credit after it has already been built can be extremely inefficient. The results of this paper uncover some fundamental difficulties for generative models to properly give credit to data.