Despite intense interest and considerable effort, the current generation of neural networks suffers a significant loss of accuracy under most practically relevant privacy training regimes. One particularly challenging class of neural networks are the wide ones, such as those deployed for NLP or recommender systems.
Observing that these models share something in common---an embedding layer that reduces the dimensionality of the input---we focus on developing a general approach towards training these models that takes advantage of the sparsity of the gradients. We propose a novel algorithm for privately training neural networks. Furthermore, we provide an empirical study of a DP wide neural network on a real-world dataset, which has been rarely explored in the previous work.
Huanyu Zhang (Facebook)
Ilya Mironov (Facebook AI)
Meisam Hejazinia (University of Texas, Dallas)
More from the Same Authors
2021 : Improved Rates for Differentially Private Stochastic Convex Optimization with Heavy-Tailed Data »
Gautam Kamath · Xingtu Liu · Huanyu Zhang
2020 : Keynote Session 4: The Shuffle Model and Federated Learning, by Ilya Mironov (Facebook) »