GUDA: Counterfactual Group-wise Training Data Attribution for Diffusion Models via Unlearning

Training-data attribution for vision generative models aims to identify which training data influenced a given output. While most methods score individual examples, practitioners often need group-level answers (e.g., artistic styles or object classes). Group-wise attribution is counterfactual: how would a model's behavior on a generated sample change if a group were absent from training? A natural realization of this counterfactual is Leave-One-Group-Out (LOGO) retraining, which retrains the model with each group removed; however, it becomes computationally prohibitive as the number of groups grows. We propose GUDA (Group Unlearning-based Data Attribution) for diffusion models, which approximates each counterfactual model by applying machine unlearning to a shared full-data model instead of training from scratch. GUDA quantifies group influence using differences in a likelihood-based scoring rule (ELBO) between the full model and each unlearned counterfactual. Experiments on CIFAR-10 and artistic style attribution with Stable Diffusion show that GUDA identifies primary contributing groups more reliably than semantic similarity, gradient-based attribution, and instance-level unlearning approaches, while achieving $\sim$100$\times$ speedup on CIFAR-10 over LOGO retraining. The code is available at [https://github.com/sony/guda](https://github.com/sony/guda).

Image-generating AI systems are trained on large collections of images, but it is often unclear which parts of the training data helped produce a particular generated image. This matters when people want to understand whether an output was influenced by a certain artistic style, object category, or other group of training examples. A direct way to answer this question is to retrain the AI system many times, each time leaving out one group, but this is usually too expensive. We propose GUDA, a more efficient way to estimate this effect. Instead of retraining from scratch, GUDA starts from the original trained model and uses a short additional training process to make the model behave as if one group had been left out. It then checks how much worse the modified model explains the generated image. If removing a group makes the image much harder to explain, that group is likely to have strongly influenced the output. In experiments on object categories and artistic styles, GUDA more reliably identified the most influential groups than methods based on visual similarity or methods that trace influence to individual training examples, while being much faster than full retraining.