Gradient Inversion Attacks Beyond SGD

Gradient Inversion Attack (GIA) poses a significant threat to federated learning, enabling adversaries to reconstruct private training data from the information shared during training. Prior research has predominantly focused on the vanilla SGD, where the server or an eavesdropper can directly observe true gradients. In practical deployments, however, models may be trained with adaptive optimizers (e.g., Adam, RMSProp, and AdaGrad), for which the observable signal is not raw gradients but momentum-based parameter updates. This setting remains underexplored and undermines traditional gradient-matching strategies, which struggle to recover labels and images from non-gradient updates. To address this gap, this paper explores attacks tailored to modern adaptive optimizers. We present an analytical rule for recovering labels from optimizer updates and propose an update-matching objective that optimizes dummy inputs to reproduce the observed updates. The proposed approach is general and can be directly applied to various optimizers such as Adam, AdaGrad, and RMSProp. Furthermore, we find that, despite being introduced for adaptive optimizers, the proposed objective function also yields stronger attacks in the standard SGD setting. Experiments on datasets such as ImageNet and PACS highlight the effectiveness of our method over existing gradient matching techniques.

Federated learning is designed to let many users train a shared machine learning model without directly sending their private data to a central server. However, previous studies have shown that the information shared during training can sometimes be used in gradient inversion attacks (GIAs) to reconstruct users’ private examples. Most existing GIAs assume that the attacker can directly observe the model’s raw gradients, while many federated learning systems train models with modern optimizers such as Adam, RMSProp, or AdaGrad, where the server may only observe processed model updates instead. This creates a gap between how GIAs are usually studied and how federated learning systems may operate in practice. Our work narrows this gap by showing that private examples can still be recovered from these model updates. To achieve this, we design a method that uses the observed updates to infer both the labels and visual content of the private training examples. This finding reveals that privacy risks in federated learning are broader than previously studied. It suggests that using modern optimizers does not, by itself, prevent private data from being reconstructed. We hope this work helps researchers design stronger protections for federated learning systems.