Timezone: »
The attack surface of machine learning is large: training data can be poisoned, predictions manipulated using adversarial examples, models exploited to reveal sensitive information contained in training data, etc. This is in large parts due to the absence of security and privacy considerations in the design of ML algorithms. Yet, adversaries have clear incentives to target these systems. Thus, there is a need to ensure that computer systems that rely on ML are trustworthy.
Fortunately, we are at a turning point where ML is still being adopted, which creates a rare opportunity to address the shortcomings of the technology before it is widely deployed. Designing secure ML requires that we have a solid understanding as to what we expect legitimate model behavior to look like. We structure our discussion around three directions, which we believe are likely to lead to significant progress.
The first encompasses a spectrum of approaches to verification and admission control, which is a prerequisite to enable fail-safe defaults in machine learning systems. The second seeks to design mechanisms for assembling reliable records of compromise that would help understand the degree to which vulnerabilities are exploited by adversaries, as well as favor psychological acceptability of machine learning applications. The third pursues formal frameworks for security and privacy in machine learning, which we argue should strive to align machine learning goals such as generalization with security and privacy desiderata like robustness or privacy. We illustrate these directions with recent work on model extraction, privacy-preserving ML and machine unlearning.
Author Information
Nicolas Papernot (University of Toronto and Vector Institute)
More from the Same Authors
-
2022 Poster: On the Difficulty of Defending Self-Supervised Learning against Model Extraction »
Adam Dziedzic · Nikita Dhawan · Muhammad Ahmad Kaleem · Jonas Guan · Nicolas Papernot -
2022 Spotlight: On the Difficulty of Defending Self-Supervised Learning against Model Extraction »
Adam Dziedzic · Nikita Dhawan · Muhammad Ahmad Kaleem · Jonas Guan · Nicolas Papernot -
2021 Poster: Markpainting: Adversarial Machine Learning meets Inpainting »
David G Khachaturov · Ilia Shumailov · Yiren Zhao · Nicolas Papernot · Ross Anderson -
2021 Poster: Label-Only Membership Inference Attacks »
Christopher Choquette-Choo · Florian Tramer · Nicholas Carlini · Nicolas Papernot -
2021 Spotlight: Label-Only Membership Inference Attacks »
Christopher Choquette-Choo · Florian Tramer · Nicholas Carlini · Nicolas Papernot -
2021 Spotlight: Markpainting: Adversarial Machine Learning meets Inpainting »
David G Khachaturov · Ilia Shumailov · Yiren Zhao · Nicolas Papernot · Ross Anderson -
2020 : Panel 1 »
Deborah Raji · Tawana Petty · Nicolas Papernot · Piotr Sapiezynski · Aleksandra Korolova -
2020 Poster: Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations »
Florian Tramer · Jens Behrmann · Nicholas Carlini · Nicolas Papernot · Joern-Henrik Jacobsen