secure multi-party computation based privacy preserving extreme learning machine algorithm over...

IntroductionPreliminaries

Privacy-preserving ELM over vertically partitioned dataExperimentsConclusions

Secure Multi-Party Computation Based PrivacyPreserving Extreme Learning Machine Algorithm

Over Vertically Distributed Data

Ferhat Ozgur [email protected]

TUBITAK - BILGEM - Cyber Security InstituteKocaeli, Turkey

Nov. 10 2015The 2015 International Data Mining and Cybersecurity Workshop

With ICONIP2015Ferhat Ozgur Catak [email protected] TUBITAK - BILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learning Machine Algorithm Over Vertically Distributed Data



Table of Contents

1 IntroductionSecure Multiparty ComputationContributions

2 PreliminariesELMSecure Multi Party ComputationSecure Multi-Party Addition

3 Privacy-preserving ELM over vertically partitioned dataPrivacy-preserving ELM

4 ExperimentsExperimental setupSimulation Results

5 ConclusionsConclusions

Ferhat Ozgur Catak [email protected] TUBITAK - BILGEM - Cyber Security Institute Kocaeli, TurkeySecure Multi-Party Computation Based Privacy Preserving Extreme Learning Machine Algorithm Over Vertically Distributed Data



Secure Multiparty ComputationContributions

Table of Contents










Secure Multiparty Computation (SMC)

DefinitionI SMC : The goal of creating methods for parties to jointly compute

a function over their inputs while keeping those inputs private.

I In an MPC,I a given number of participants, p1, p2, · · · , pNI each has private data, respectively d1, d2, · · · , dN

I Participants want to compute the value of a public function onthat private data:

I F (d1, d2, ..., dN) while keeping their own inputs secret.I Challenges :

I To protect each participant’s private data, and intermediate resultsI The computation/communication cost introduced to each

participant shall be affordableI Training data is arbitrarily partitioned





Contributions

Main contributions of the workI Privacy-preserving Extreme Learning Machine (ELM) training

modelI Global ELM classification model from the distributed data sets in

multiple partiesI Training data set is vertically partitioned among the partiesI The final distributed model is constructed at an independent party




ELMSecure Multi Party ComputationSecure Multi-Party Addition

Table of Contents










Extreme Learning Machine (ELM)

Learning Without Iterative TuningI All hidden node parameters can be randomly generated without the

knowledge of the training data. That is, for any continuous targetfunction f and any randomly generated sequencelimL→∞ ||f (x)− fL(x)|| = limL→∞ ||f (x)−

∑Li=1 βiG(ai , bi , x)|| = 0

holds with probability one if βi is chosen to minimize||f (x)− fL(x)||,∀i

G.-B. Huang, et al., ”Universal approximation using incremental constructivefeedforward networks with random hidden nodes,” IEEE Transactions on NeuralNetworks, vol. 17, no. 4, pp. 879-892, 2006.

G.-B. Huang, et al., ”Convex incremental extreme learning machine,”Neurocomputing, vol. 70, pp. 3056-3062, 2007.

G.-B. Huang, et al., ”Enhanced random search based incremental extreme learningmachine,” Neurocomputing, vol. 71, pp. 3460-3468, 2008.






Figure: Generalized SLFNMathematical Model

I For N arbitrary distinct samples(xi , ti ) ∈ Rn × Rm, SLFNs with Lhidden nodes each with outputfunction, G(ai , bi , x) aremathematically modeled as

L∑i=1

βi G(ai , bi , xj ) = tj , j = 1, ...,N

(1)I (ai , bi ): hidden node parameters.I βi : the weight vector connecting the

ith hidden node and the output node.






Mathematical Model∑Li=1 βiG(ai , bi , xj) = tj , j = 1, ...,N is equivalent to Hβ = T where,h(x1)

...h(xN)

=

G(a1, b1, x1) · · · G(aL, bL, x1)...

. . ....

G(a1, b1, xN) · · · G(aL, bL, xN)

N×L

(2)

β =

βT1...βT

L

L×m

ve T =

tT1...

tTN

N×m

(3)

H, is called the hidden layer output matrix of the neural network, the ithcolumn of H is the output of the ith hidden node with respect to inputs ,x1, ..., xn






Three-Step Learning ModelGiven a training set D = {(xi , ti )|xi ∈ Rn, ti ∈ Rm, i = 1, · · · ,N}, hiddennode output function G(a, b, x), and the number of hidden nodes L,

1 Assign randomly hidden node parameters (ai , bi ), i = 1, · · · , L.2 Calculate the hidden layer output matrix H.3 Calculate the output weight β = HtT.





Secure Multi Party Computation

I Secure Multi-Party ComputationI The problem of n players to compute an agreed function f of their

inputs

In vertically partitioned data, each party holds different attributes ofsame data set.The partition strategy is shown in Figure 2.

x1,1 · · · x1,t−1 · · · x1,t · · · x1,k

x2,1 · · · x2,t−1 · · · x2,t · · · x2,k

......

......

......

...xm,1 · · · xm,t−1 · · · xm,t · · · xm,k

P1 · · · Pk

D =

Figure: Vertically partitioned data set D.





Secure Multi-Party Addition I

I Secure Multi-Party AdditionI Assume k ≥ 3 parties, P0, · · · , Pk−1, with party Pi holding value viI Aim is to compute v =

∑k−1i=0 vi

I Sum of vi lies in FI P0 randomly chooses a number R ∈ F

I P0 adds R to its local value v0,I Sends v0 = R + v0mod |F | to P1

I The remaining parties Pi , i = 1, · · · , k − 1I Pi receives V = R +

∑i−1j=0 vjmod |F|

I Pi sends R +∑

j=1 ivjmod |F| = (vi + V )mod |F|I Finally, P0, subtracts R from the final message to compute actual

results.





Secure Multi-Party Addition II

Party0x0 , R ∈ F

Party1x1

Party2x2

Partykxk

V = R+ x0mod|F |

V = R+∑i−1

j=0 xjmod|F|

(xi + V )mod|F|

Figure: Secure Multi-Party Addition.





Secure Multi-Party Addition III

Algorithm 1: Secure multi-party addition1: procedure SMA(P)2: P0 : R ← rand(F) . P0 randomly chooses a number R3: V ← R + x0 mod F4: P0 sends V to node P15: for i = 1, · · · , k − 1 do6: Pi receives V = R +

∑i−1j=0

xj mod F

7: Pi computes V =(

R +∑i

j=1xj mod F

)= ((xi + V ) mod F)

8: Pi sends V to node Pi+1

9: P0 : V ← (V − R) = (V − R mod F) . Actual addition result10: end procedure




Privacy-preserving ELM

Table of Contents










Privacy-preserving ELM I

1 Master party creates weight matrix, W ∈ RL×N

2 Master party distributes partition W with same feature size for eachparties.

3 Party P0 creates a random matrix,

R =

rand1,1(F) · · · rand1,L(F)...

. . ....

randN,1(F) · · · randN,L(F)

N×L

4 Party P0 creates perturbated output,

V = R +

(x0

1 ·w01 + b0

1)· · ·

(x0

1 ·w0L + b0

L)

.... . .

...(x0

N ·w01 + b0

1)· · ·

(x0

N ·w0L + b0

L)

5 for i = 1, · · · , k − 1





Privacy-preserving ELM II

I Pi computes V = V +

(xi

1 · wi1 + bi

1)· · ·

(xi

1 · wiL + bi

L)

.... . .

...(xi

N · wi1 + bi

1)· · ·

(xi

L · wiN + bi

N)

I Pi sends V to Pi+1

6 P0 subtracts random matrix, R, from the received matrix V.H = (V− R) mod F

7 Hidden layer node weight vector, β, is calculated. β = H† · T





Privacy-preserving ELM III

Party0

x0, w, H0

,R ∈ F ,

Party1x1, H1, w

Party2x2, H2, w

Partykxk−1,

Hk−1, w

V0 = R +H0mod|F |

V1 = V0+H1mod|F |

V2 = V1 +H2mod|F |

Vk = Vk−1 +Hkmod|F |

Hi =

G(w1, b1,xi1) · · · G(wL, bL,xi

1)

.

.

.. . .

.

.

.

G(w1, b1,xiN ) · · · G(wL, bL,xi

N )

N×L

R =

rand1,1(F) · · · rand1,L(F)

.

.

.. . .

.

.

.randN,1(F) · · · randN,L(F)

N×L

H = (V −R) mod F

Figure: Overview of privacy-preserving ELM learning.




Experimental setupSimulation Results

Table of Contents










Experimental setup I

I Our approach is applied to six different data sets to verify its modeleffectivity and efficiency.

Table: Description of the testing data sets used in the experiments.

Data set #Train #Classes #Attributesaustralian 690 2 14

colon-cancer 62 2 2,000diabetes 768 2 8

duke breast cancer 44 2 7,129heart 270 2 13

ionosphere 351 2 34





Simulation Results I

I The number of party size, k: from 3 to number of feature n, of thedata set

I For instance, k = 3, and n = 14, then the first two party have 5attributes, and last party has 4 attributes.

I The accuracy of secure multi-party computation based ELM isexactly same for the traditional ELM training algorithm.





Simulation Results III Figure shows results of our simulations.I Time scale becomes its steady state position when number of parties

I k, moves closer to number of attributes, n.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

10−1

100

101

102

103

Attr ibu teS iz e(n)P ar tyS iz e(k )

Tim

e s

ca

le

australian colon−cancer diabetes duke heart ionosphere

Figure: Vertically partitioned data set D.




Conclusions

Table of Contents









Conclusions

Conclusions

Summary

I ELM learning algorithm, a new method.I ELM outperforms traditional Single Layer Feed-forward Neural-networks and

Support Vector Machines for Big Data 1

I In all fields that ELM is applied (i.e. medical records, business, government),privacy is a major concern.

Conclusions

I Privacy-preserving learning model is proposed in vertically partitioned dataI In multi-party partitioning without sharing the data of each site to

the others.I Master party divides weight vector, and each party calculates the activation

function result with its data and weight vector.

1Cambria, Erik, et al. ”Extreme learning machines [trends & controversies].”Intelligent Systems, IEEE 28.6 (2013): 30-59.




Conclusions

Thank YouFerhat Ozgur Catak

[email protected]


secure multi-party computation based privacy preserving extreme learning machine algorithm over...

Data & Analytics