Timezone: »

 
Spotlight
Public Data-Assisted Mirror Descent for Private Model Training
Ehsan Amid · Arun Ganesh · Rajiv Mathews · Swaroop Ramaswamy · Shuang Song · Thomas Steinke · Thomas Steinke · Vinith Suriyakumar · Om Thakkar · Abhradeep Guha Thakurta

Tue Jul 19 08:45 AM -- 08:50 AM (PDT) @ Ballroom 1 & 2
in Social Aspects »

In this paper, we revisit the problem of using in-distribution public data to improve the privacy/utility trade-offs for differentially private (DP) model training. (Here, public data refers to auxiliary data sets that have no privacy concerns.) We design a natural variant of DP mirror descent, where the DP gradients of the private/sensitive data act as the linear term, and the loss generated by the public data as the mirror map.We show that, for linear regression with feature vectors drawn from a non-isotropic sub-Gaussian distribution, our algorithm, PDA-DPMD (a variant of mirror descent), provides population risk guarantees that are asymptotically better than the best known guarantees under DP (without having access to public data), when the number of public data samples is sufficiently large. We further show that our algorithm has natural ``noise stability'' properties that control the variance due to noise added to ensure DP.We demonstrate the efficacy of our algorithm by showing privacy/utility trade-offs on four benchmark datasets (StackOverflow, WikiText-2, CIFAR-10, and EMNIST). We show that our algorithm not only significantly improves over traditional DP-SGD, which does not have access to public data, but to our knowledge is the first to improve over DP-SGD on models that have been pre-trained with public data.

Keywords: DL: Everything Else · T: Deep Learning · T: Optimization · SA: Privacy-preserving Statistics and Machine Learning

Author Information

Ehsan Amid (Google Brain)
Arun Ganesh (UC Berkeley)
Rajiv Mathews (Google)
Swaroop Ramaswamy (Google)
Shuang Song (Google)
Thomas Steinke (Google)
Thomas Steinke (Google)
Vinith Suriyakumar (Massachusetts Institute of Technology)
Om Thakkar (Google)
Abhradeep Guha Thakurta (Google)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors