pennsylvania state university & google brain nicolas...
TRANSCRIPT
![Page 1: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/1.jpg)
Security and Privacyin Machine LearningNicolas PapernotPennsylvania State University & Google Brain
Lecture for Prof. Trent Jaeger’s CSE 543 Computer Security Class
November 2017 - Penn State
![Page 2: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/2.jpg)
Patrick McDaniel(Penn State)
Ian Goodfellow(Google Brain)
Martín Abadi (Google Brain)Pieter Abbeel (Berkeley)Michael Backes (CISPA)Dan Boneh (Stanford)Z. Berkay Celik (Penn State)Yan Duan (OpenAI)Úlfar Erlingsson (Google Brain)Matt Fredrikson (CMU)Kathrin Grosse (CISPA)Sandy Huang (Berkeley)Somesh Jha (U of Wisconsin)
Thank you to my collaborators
2
Alexey Kurakin (Google Brain)Praveen Manoharan (CISPA)Ilya Mironov (Google Brain)Ananth Raghunathan (Google Brain)Arunesh Sinha (U of Michigan) Shuang Song (UCSD)Ananthram Swami (US ARL)Kunal Talwar (Google Brain)Florian Tramèr (Stanford)Michael Wellman (U of Michigan)Xi Wu (Google)
![Page 3: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/3.jpg)
3
Machine Learning Classifier
[0.01, 0.84, 0.02, 0.01, 0.01, 0.01, 0.05, 0.01, 0.03, 0.01]
[p(0|x,θ), p(1|x,θ), p(2|x,θ), …, p(7|x,θ), p(8|x,θ), p(9|x,θ)] f(x,θ)x
Classifier: map inputs to one class among a predefined set
![Page 4: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/4.jpg)
4
Machine Learning Classifier
[0 1 0 0 0 0 0 0 0 0][0 1 0 0 0 0 0 0 0 0]
[1 0 0 0 0 0 0 0 0 0][0 0 0 0 0 0 0 1 0 0]
[0 0 0 0 0 0 0 0 0 1][0 0 0 1 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 1 0][0 0 0 0 0 0 1 0 0 0]
[0 1 0 0 0 0 0 0 0 0][0 0 0 0 1 0 0 0 0 0]
Learning: find internal classifier parameters θ that minimize a cost/loss function (~model error)
![Page 5: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/5.jpg)
Outline of this lecture
5
1
2
Security in ML
Privacy in ML
![Page 6: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/6.jpg)
Part I
Security in machine learning
6
![Page 7: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/7.jpg)
Attacker may see the model: bad even if an attacker needs to know details of the machine
learning model to do an attack --- aka a white-box attacker
Attacker may not need the model: worse if attacker who knows very little (e.g. only gets to
ask a few questions) can do an attack --- aka a black-box attacker
Attack Models
7
ML
ML
Papernot et al. Towards the Science of Security and Privacy in Machine Learning
![Page 8: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/8.jpg)
Attacker may see the model: bad even if an attacker needs to know details of the machine
learning model to do an attack --- aka a white-box attacker
Attacker may not need the model: worse if attacker who knows very little (e.g. only gets to
ask a few questions) can do an attack --- aka a black-box attacker
Attack Models
8
ML
ML
Papernot et al. Towards the Science of Security and Privacy in Machine Learning
![Page 9: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/9.jpg)
Adversarial examples(white-box attacks)
9
![Page 10: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/10.jpg)
Jacobian-based Saliency Map Approach (JSMA)
10Papernot et al. The Limitations of Deep Learning in Adversarial Settings
![Page 11: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/11.jpg)
11
Jacobian-Based Iterative Approach: source-target misclassification
Papernot et al. The Limitations of Deep Learning in Adversarial Settings
![Page 12: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/12.jpg)
Evading a Neural Network Malware Classifier
DREBIN dataset of Android applications
Add constraints to JSMA approach: - only add features: keep malware behavior - only features from manifest: easy to modify
“Most accurate” neural network- 98% accuracy, with 9.7% FP and 1.3% FN - Evaded with a 63.08% success rate
12Grosse et al. Adversarial Perturbations Against Deep Neural Networks for Malware Classification
P[X=Malware] = 0.90P[X=Benign] = 0.10
P[X*=Malware] = 0.10P[X*=Benign] = 0.90
![Page 13: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/13.jpg)
Supervised vs. reinforcement learning
13
Supervised learning Reinforcement learning
Model inputs Observation (e.g., traffic sign, music, email) Environment & Reward function
Model outputsClass
(e.g., stop/yield, jazz/classical, spam/legitimate)
Action
Training “goal”(i.e., cost/loss)
Minimize class prediction errorover pairs of (inputs, outputs)
Maximize reward by exploring the environment and
taking actions
Example
![Page 14: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/14.jpg)
Adversarial attacks on neural network policies
14Huang et al. Adversarial Attacks on Neural Network Policies
![Page 15: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/15.jpg)
Adversarial examples(black-box attacks)
15
![Page 16: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/16.jpg)
Threat model of a black-box attack
Training dataModel architecture Model parameters
Model scores
Adversarial capabilities
(limited) oracle access: labels
Adversarial goal Force a ML model remotely accessible through an API to misclassify
16
Example
![Page 17: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/17.jpg)
Our approach to black-box attacks
17
Alleviate lack of knowledge about model
Alleviate lack of training data
![Page 18: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/18.jpg)
Adversarial example transferability
Adversarial examples have a transferability property:
samples crafted to mislead a model A are likely to mislead a model B
These property comes in several variants:
● Intra-technique transferability:○ Cross model transferability○ Cross training set transferability
● Cross-technique transferability
18
ML A
Szegedy et al. Intriguing properties of neural networks
![Page 19: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/19.jpg)
Adversarial examples have a transferability property:
samples crafted to mislead a model A are likely to mislead a model B
These property comes in several variants:
● Intra-technique transferability:○ Cross model transferability○ Cross training set transferability
● Cross-technique transferability
19Szegedy et al. Intriguing properties of neural networks
ML A
ML B Victim
Adversarial example transferability
![Page 20: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/20.jpg)
Adversarial examples have a transferability property:
samples crafted to mislead a model A are likely to mislead a model B
20
Adversarial example transferability
![Page 21: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/21.jpg)
Cross-technique transferability
21Papernot et al. Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
![Page 22: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/22.jpg)
Cross-technique transferability
22Papernot et al. Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
![Page 23: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/23.jpg)
Our approach to black-box attacks
23
Adversarial example transferability from a substitute model to
target model
Alleviate lack of knowledge about model
Alleviate lack of training data
![Page 24: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/24.jpg)
Attacking remotely hosted black-box models
24
Remote ML sys
“no truck sign”“STOP sign”
“STOP sign”
(1) The adversary queries remote ML system for labels on inputs of its choice.
![Page 25: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/25.jpg)
25
Remote ML sys
Local substitute
“no truck sign”“STOP sign”
“STOP sign”
(2) The adversary uses this labeled data to train a local substitute for the remote system.
Attacking remotely hosted black-box models
![Page 26: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/26.jpg)
26
Remote ML sys
Local substitute
“no truck sign”“STOP sign”
(3) The adversary selects new synthetic inputs for queries to the remote ML system based on the local substitute’s output surface sensitivity to input variations.
Attacking remotely hosted black-box models
![Page 27: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/27.jpg)
27
Remote ML sys
Local substitute
“yield sign”
(4) The adversary then uses the local substitute to craft adversarial examples, which are misclassified by the remote ML system because of transferability.
Attacking remotely hosted black-box models
![Page 28: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/28.jpg)
Our approach to black-box attacks
28
Adversarial example transferability from a substitute model to
target model
Synthetic data generation
+
Alleviate lack of knowledge about model
Alleviate lack of training data
![Page 29: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/29.jpg)
Results on real-world remote systems
29
All remote classifiers are trained on the MNIST dataset (10 classes, 60,000 training samples)
Remote Platform ML technique Number of queriesAdversarial examples
misclassified (after querying)
Deep Learning 6,400 84.24%
Logistic Regression 800 96.19%
Unknown 2,000 97.72%
[PMG16a] Papernot et al. Practical Black-Box Attacks against Deep Learning Systems using Adversarial Examples
![Page 30: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/30.jpg)
Benchmarking progress in the adversarial MLcommunity
30
![Page 31: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/31.jpg)
31
![Page 32: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/32.jpg)
32
Growing community
1.3K+ stars340+ forks
40+ contributors
![Page 33: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/33.jpg)
33
Adversarial examples represent worst-case distribution drifts
[DDS04] Dalvi et al. Adversarial Classification (KDD)
![Page 34: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/34.jpg)
34
Adversarial examples are a tangible instance of hypothetical AI safety problems
Image source: http://www.nerdist.com/wp-content/uploads/2013/07/Space-Odyssey-4.jpg
![Page 35: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/35.jpg)
Part II
Privacy in machine learning
35
![Page 36: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/36.jpg)
Types of adversaries and our threat model
36
In our work, the threat model assumes:
- Adversary can make a potentially unbounded number of queries- Adversary has access to model internals
Model inspection (white-box adversary)Zhang et al. (2017) Understanding DL requires rethinking generalization
Model querying (black-box adversary)Shokri et al. (2016) Membership Inference Attacks against ML ModelsFredrikson et al. (2015) Model Inversion Attacks
?Black-box
ML
![Page 37: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/37.jpg)
A definition of privacy
37
Randomized Algorithm
Randomized Algorithm
Answer 1Answer 2
...Answer n
Answer 1Answer 2
...Answer n
????}
![Page 38: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/38.jpg)
Our design goals
38
Preserve privacy of training data when learning classifiers
Differential privacy protection guarantees
Intuitive privacy protection guarantees
Generic* (independent of learning algorithm)Goals
Problem
*This is a key distinction from previous work, such as Pathak et al. (2011) Privacy preserving probabilistic inference with hidden markov models Jagannathan et al. (2013) A semi-supervised learning approach to differential privacy Shokri et al. (2015) Privacy-preserving Deep Learning Abadi et al. (2016) Deep Learning with Differential Privacy Hamm et al. (2016) Learning privately from multiparty data
![Page 39: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/39.jpg)
The PATE approach
39
![Page 40: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/40.jpg)
Teacher ensemble
40
Partition 1
Partition 2
Partition n
Partition 3
...
Teacher 1
Teacher 2
Teacher n
Teacher 3
...
Training
Sensitive Data
Data flow
![Page 41: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/41.jpg)
Aggregation
41
Count votes Take maximum
![Page 42: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/42.jpg)
Intuitive privacy analysis
42
If most teachers agree on the label, it does not depend on specific partitions, so the privacy cost is small.
If two classes have close vote counts, the disagreement may reveal private information.
![Page 43: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/43.jpg)
Noisy aggregation
43
Count votes Add Laplacian noise Take maximum
![Page 44: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/44.jpg)
Teacher ensemble
44
Partition 1
Partition 2
Partition n
Partition 3
...
Teacher 1
Teacher 2
Teacher n
Teacher 3
...
Aggregated Teacher
Training
Sensitive Data
Data flow
![Page 45: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/45.jpg)
Student training
45
Partition 1
Partition 2
Partition n
Partition 3
...
Teacher 1
Teacher 2
Teacher n
Teacher 3
...
Aggregated Teacher Student
Training
Available to the adversaryNot available to the adversary
Sensitive Data
Public Data
Inference Data flow
Queries
![Page 46: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/46.jpg)
Why train an additional “student” model?
46
Each prediction increases total privacy loss.Privacy budgets create a tension between the accuracy and number of predictions.
Inspection of internals may reveal private data.Privacy guarantees should hold in the face of white-box adversaries.
1
2
The aggregated teacher violates our threat model:
![Page 47: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/47.jpg)
Student training
47
Partition 1
Partition 2
Partition n
Partition 3
...
Teacher 1
Teacher 2
Teacher n
Teacher 3
...
Aggregated Teacher Student
Training
Available to the adversaryNot available to the adversary
Sensitive Data
Public Data
Inference Data flow
Queries
![Page 48: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/48.jpg)
Deployment
48
Inference
Available to the adversary
QueriesStudent
![Page 49: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/49.jpg)
Differential privacy:A randomized algorithm M satisfies ( , ) differential privacy if for all pairs of neighbouring datasets (d,d’), for all subsets S of outputs:
Application of the Moments Accountant technique (Abadi et al, 2016)
Strong quorum ⟹ Small privacy cost
Bound is data-dependent: computed using the empirical quorum
Differential privacy analysis
49
![Page 50: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/50.jpg)
Experimental results
50
![Page 51: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/51.jpg)
Experimental setup
51
Dataset Teacher Model Student Model
MNIST Convolutional Neural Network Generative Adversarial Networks
SVHN Convolutional Neural Network Generative Adversarial Networks
UCI Adult Random Forest Random Forest
UCI Diabetes Random Forest Random Forest
/ /models/tree/master/differential_privacy/multiple_teachers
![Page 52: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/52.jpg)
Aggregated teacher accuracy
52
![Page 53: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/53.jpg)
Trade-off between student accuracy and privacy
53
![Page 54: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/54.jpg)
Trade-off between student accuracy and privacy
54
UCI Diabetes
1.44
10-5
Non-private baseline 93.81%
Student accuracy
93.94%
![Page 55: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/55.jpg)
Synergy between privacy and generalization
55
![Page 56: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/56.jpg)
www.papernot.fr56@NicolasPapernot
Some online ressources:
Blog on S&P in ML (joint work w/ Ian Goodfellow) www.cleverhans.ioML course https://coursera.org/learn/machine-learningDL course https://coursera.org/learn/neural-networks
Assigned reading and more in-depth technical survey paper:
Machine Learning in Adversarial SettingsPatrick McDaniel, Nicolas Papernot, Z. Berkay Celik
Towards the Science of Security and Privacy in Machine LearningNicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael Wellman
![Page 57: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/57.jpg)
57
![Page 58: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/58.jpg)
Gradient masking
58Tramèr et al. Ensemble Adversarial Training: Attacks and Defenses
![Page 59: Pennsylvania State University & Google Brain Nicolas Papernottrj1/cse543-f17/slides/papernot-adv-ml.pdf · 2017-11-14 · (Google Brain) Martín Abadi (Google Brain) Pieter Abbeel](https://reader033.vdocuments.us/reader033/viewer/2022050409/5f8633b90756f108695884b8/html5/thumbnails/59.jpg)
Gradient masking
59Tramèr et al. Ensemble Adversarial Training: Attacks and Defenses