Timezone: »

 
Spotlight
Private Adaptive Gradient Methods for Convex Optimization
Hilal Asi · John Duchi · Alireza Fallah · Omid Javidbakht · Kunal Talwar

Thu Jul 22 06:40 PM -- 06:45 PM (PDT) @
in Privacy 4 »

We study adaptive methods for differentially private convex optimization, proposing and analyzing differentially private variants of a Stochastic Gradient Descent (SGD) algorithm with adaptive stepsizes, as well as the AdaGrad algorithm. We provide upper bounds on the regret of both algorithms and show that the bounds are (worst-case) optimal. As a consequence of our development, we show that our private versions of AdaGrad outperform adaptive SGD, which in turn outperforms traditional SGD in scenarios with non-isotropic gradients where (non-private) Adagrad provably outperforms SGD. The major challenge is that the isotropic noise typically added for privacy dominates the signal in gradient geometry for high-dimensional problems; approaches to this that effectively optimize over lower-dimensional subspaces simply ignore the actual problems that varying gradient geometries introduce. In contrast, we study non-isotropic clipping and noise addition, developing a principled theoretical approach; the consequent procedures also enjoy significantly stronger empirical performance than prior approaches.

Keywords: Privacy, Anonymity, and Security · Social Aspects of Machine Learning

Author Information

Hilal Asi (Stanford University)
John Duchi (Stanford University)
Alireza Fallah (MIT)
Omid Javidbakht (Apple)
Kunal Talwar (Apple)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors