the data streaming paradigm and its use in fog architectures

85
The data streaming paradigm and its use in Fog architectures Vincenzo Gulisano, Ph.D. EBSIS Summer School 2016 1

Upload: vincenzo-gulisano

Post on 22-Jan-2018

571 views

Category:

Data & Analytics


2 download

TRANSCRIPT

ThedatastreamingparadigmanditsuseinFogarchitectures

VincenzoGulisano,Ph.D.

EBSISSummerSchool2016

1

Agenda

• Introduction:Fogarchitectures• Datastreamingbasics• Trade-offsandchallengesinFogstreaminganalysis• Conclusions• Bibliography

2ThedatastreamingparadigmanditsuseinFogarchitectures

Agenda

• Introduction:Fogarchitectures• Datastreamingbasics• Trade-offsandchallengesinFogstreaminganalysis• Conclusions• Bibliography

3ThedatastreamingparadigmanditsuseinFogarchitectures

Introduction:Fogarchitectures

AdvancedMetering Infrastructures(AMIs)[1,2,3,4]

VehicularNetworks(VNs)[5,6]

4ThedatastreamingparadigmanditsuseinFogarchitectures

AMIs VNs

• demand-response• scheduling [7]• micro-grids• detectionofmediumsizeblackouts [8]• detectionofnon technicallosses• ...

5

• autonomous driving• platooning• accidentdetection[9]• variabletolls[9]• congestionmonitoring [10]• ...

Accesstotheirdataà Increasedawareness,security,power-efficiency,...

ThedatastreamingparadigmanditsuseinFogarchitectures

(some)fundamentalquestions

• wheredoweprocessdata?• howdoweprocessdata?• howdoweenforcesecurityandprivacy?

6ThedatastreamingparadigmanditsuseinFogarchitectures

thewrong incompleteanswerweruntheanalysisinthecloud!

7

1. doestheinfrastructureallowforbillionsofreadingsperdaytobetransferredcontinuously?

2. thelatencyincurredwhiletransferringdata,doesthatunderminetheutilityoftheanalysis?

3. isitsecuretoconcentrateallthedatainasingleplace?[11]

4. isitsmarttogiveawayfine-graineddata?[12]

ThedatastreamingparadigmanditsuseinFogarchitectures

Asmallexampleofwhatfine-graineddatacanreveal...

8

source:PrivateMemoirsofaSmartMeter[12]

ThedatastreamingparadigmanditsuseinFogarchitectures

abetteranswerweleveragetheentireinfrastructure!

9

Traditionalanalysistechniquescannotaddressallthechallengesinthesesetups[13,14]

ThedatastreamingparadigmanditsuseinFogarchitectures

Fogarchitectures

10

source:FogComputingandItsRoleintheInternetofThings[15]

Characteristics[15]:1. edgelocation/location

awareness/lowlatency2. geographicaldistribution3. large-scale4. supportformobility5. real-timeinteractions6. predominanceofwireless7. heterogeneous8. interoperability/federation9. interactionwiththecloud

ThedatastreamingparadigmanditsuseinFogarchitectures

Fogarchitectures,applicationstrade-offsandchallenges

11

Sensor/EdgeDevices FogDevices CloudDevices

Cost + ++ +++1

Computationalpower + ++ +++

Inter-nodeheterogeneity +++ ++ +

Intra-node heterogeneity + + +++

Size +++ ++ +

Communication wireless wireless/wired wired

Volumeofdata millionsofsmallstreams2 thousands ofmediumaggregatedstreams manylargestreams

Evolutionovertime +++ ++ +3

1. depends onwhoownsthephysical hardware2. withexceptionslikee.g.Lidarsensors [16,17]3. Virtualenvironments canalsoevolve ”behindthescenes”

ThedatastreamingparadigmanditsuseinFogarchitectures

Agenda

• Introduction:Fogarchitectures• Datastreamingbasics• Trade-offsandchallengesinFogstreaminganalysis• Conclusions• Bibliography

12ThedatastreamingparadigmanditsuseinFogarchitectures

Motivation

Applications insensornetworks,cyber-physicalsystems:• largeandfluctuatingvolumesofdatagenerated

continuouslydemand for:• Continuous processingofdatastreams• Inareal-timefashion

Store-then-processisnotfeasible!!!

13ThedatastreamingparadigmanditsuseinFogarchitectures

MainMemory

Motivation

DBMSvs.DSMS

Disk

1 Data

QueryProcessing

3 Queryresults

2 Query

MainMemory

QueryProcessing

ContinuousQueryData Query

results

14ThedatastreamingparadigmanditsuseinFogarchitectures

Beforewestart...aboutdatastreamingandStreamProcessingEngines(SPEs)

15

Anincomplete,non-sortedlistofSPEs(cf.relatedworkin[18]):

time

BorealisThe Aurora Project

STanfordstREamdatAManager

NiagaraCQ

COUGAR

StreamCloud

Coveringallofthem/discussingwhichusecasesarebestforeachoneoutofscope...willfocusontheonesIworkwithandonexamplefromvehicularnetworks

ThedatastreamingparadigmanditsuseinFogarchitectures

datastream:unboundedsequenceoftuplessharingthesameschema

16

Example:vehicles’speedreports

time

Field Field

vehicleid text

time(secs) text

speed(Km/h) double

Xcoordinate double

Ycoordinate double

A 8:00 55.5 X1 Y1

Let’sassumeeachsource(e.g.,vehicle)producesanddeliversatimestampsortedstream

A 8:07 34.3 X3 Y3

A 8:03 70.3 X2 Y2

ThedatastreamingparadigmanditsuseinFogarchitectures

continuousquery(orsimplyquery):DirectedAcyclicGraph(DAG)ofstreamsandoperators

17

OP

OP

OP

OP OP

OP OP

sourceop(1+outstreams)

sinkop(1+instreams)

stream

op(1+in,1+outstreams)

ThedatastreamingparadigmanditsuseinFogarchitectures

datastreamingoperators

Twomaintypes:• Statelessoperators

• donotmaintainanystate• one-by-oneprocessing• iftheymaintainsomestate,suchstatedoesnotevolvedependingonthetuplesbeingprocessed

• Statefuloperators• maintainastatethatevolvesdependingonthetuplesbeingprocessed

• produceoutputtuplesthatdependonmultipleinputtuples

18

OP

OP

ThedatastreamingparadigmanditsuseinFogarchitectures

statelessoperators

19

Filter ...

Map

Union...

Filter/routetuplesbasedonone(ormore)conditions

Transformeachtuple

Mergemultiplestreams(withthesameschema)intoone

ThedatastreamingparadigmanditsuseinFogarchitectures

statelessoperators

20

Filter ...

Map

Union...

source:http://storm.apache.org/releases/2.0.0-SNAPSHOT/Trident-tutorial.html

ThedatastreamingparadigmanditsuseinFogarchitectures

statelessoperators

21

Filter ...

Map

Union...

source:https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html

ThedatastreamingparadigmanditsuseinFogarchitectures

statelessoperators

22

Filter ...

Map

Union...

source:http://spark.apache.org/docs/latest/streaming-programming-guide.html#transformations-on-dstreams

ThedatastreamingparadigmanditsuseinFogarchitectures

statefuloperators

23

Aggregateinformationfrommultipletuples(e.g.,max,min,sum,...)

Jointuplescomingfrom2streamsgivenacertainpredicate

Aggregate

Join

ThedatastreamingparadigmanditsuseinFogarchitectures

statefuloperators

24

source:http://storm.apache.org/releases/2.0.0-SNAPSHOT/Trident-tutorial.html

source:http://spark.apache.org/docs/latest/streaming-programming-guide.html#transformations-on-dstreams

source:http://spark.apache.org/docs/latest/streaming-programming-guide.html#transformations-on-dstreams

ThedatastreamingparadigmanditsuseinFogarchitectures

Waitamoment!

ifstreamsareunbounded,howcanweaggregateorjoin?

25ThedatastreamingparadigmanditsuseinFogarchitectures

windows andstatefulanalysis[18]

Statefuloperationsaredoneoverwindows:• Time-based(e.g.,tuplesinthelast10minutes)• Tuple-based(e.g.,giventhelast50tuples)

26

time[8:00,9:00)

[8:20,9:20)

[8:40,9:40)

Usuallyapplicationsrelyontime-basedslidingwindows

ThedatastreamingparadigmanditsuseinFogarchitectures

time-basedslidingwindowaggregation(count)

27

Counter:4

time[8:00,9:00)

8:05 8:15 8:22 8:45 9:05

Output:4

Counter:1Counter:2

Counter:3

Counter:3

time

8:05 8:15 8:22 8:45 9:05

[8:20,9:20)

weassumedeachsourceproducesanddeliversatimestampsortedstream!Whathappensifthisisnotthecase?

ThedatastreamingparadigmanditsuseinFogarchitectures

time-basedslidingwindowjoining

28

t1

t2

t3

t4

t1

t2

t3

t4

R S

Slidingwindow Window

size WSWSWR

PredicateP

ThedatastreamingparadigmanditsuseinFogarchitectures

29

windows andstatefulanalysis

ThedatastreamingparadigmanditsuseinFogarchitectures

30

basicoperatorsanduser-definedoperators

Besidesasetofbasicoperators,SPEsusuallyallowtheusertodefinead-hocoperators(e.g.,whenexistingaggregationarenotenough)

ThedatastreamingparadigmanditsuseinFogarchitectures

samplequery

Foreachvehicle,raiseanalertifthespeedofthelatestreportismorethan2timeshigherthanitsaveragespeedinthelast30days.

31

time

A 8:00 55.5 X1 Y1 A 8:07 34.3 X3 Y3

A 8:03 70.3 X2 Y2

ThedatastreamingparadigmanditsuseinFogarchitectures

32

Removeunusedfields

Map

Field

vehicleid

time(secs)

speed(Km/h)

Xcoordinate

Ycoordinate

Field

vehicleid

time(secs)

speed(Km/h)

Computeaveragespeedforeach

vehicleduring thelast30days

Aggregate

Fieldvehicleid

time(secs)

avgspeed(Km/h)

Join

Checkcondition

Filter

Field

vehicleid

time(secs)

speed(Km/h)

Joinonvehicleid

Fieldvehicleid

time(secs)

avgspeed(Km/h)

speed(Km/h)

samplequery

ThedatastreamingparadigmanditsuseinFogarchitectures

33

M A J F

samplequery

Notice:• thesamesemanticscanbedefinedinseveralways(usingdifferentoperatorsandcomposingthemindifferentways)

• Usingmanybasicbuildingblockscaneasethetaskofdistributingandparallelizingtheanalysis(moreinthefollowing...)

ThedatastreamingparadigmanditsuseinFogarchitectures

Agenda

• Introduction:Fogarchitectures• Datastreamingbasics• Trade-offsandchallengesinFogstreaminganalysis• Conclusions• Bibliography

34ThedatastreamingparadigmanditsuseinFogarchitectures

1. Distributeddeployment2. Paralleldeployment3. Orderinganddeterminism4. Shared-nothingvsshared-memoryparallelism5. Loadbalancing6. Elasticity7. Faulttolerance8. Datasharing(differentialprivacy)

35

8 aspects/usecasestodiscusschallenges(and opportunities)

ThedatastreamingparadigmanditsuseinFogarchitectures

Beforewestart...

36

Followingexamplesarefromvehicularnetworks

Road-sideunitRSU Vehicle

Server

ThedatastreamingparadigmanditsuseinFogarchitectures

1- Distributeddeployment– wheretoplaceagivenoperator?[19]

37

M

1. Whathardwareisavailableateachpossibledeploymentlocation?

2. Whatdataisavailableateachpossibledeploymentlocation,dependingon1. theinfrastructure?2. itssecurityregulations?3. itsprivacyregulations?

?

ThedatastreamingparadigmanditsuseinFogarchitectures

1- Distributeddeployment– wheretoplaceagivenoperator?

38

M1. Lowlatency(ifalltherequireddataislocally

accessible)2. Accesstoalllocaldata(partsofwhichmight

notbereportedtoRSUs/Servers)[4]3. Embeddeddevices/limitedcomputational

power[20]

ThedatastreamingparadigmanditsuseinFogarchitectures

1- Distributeddeployment– wheretoplaceagivenoperator?

39

M1. Accesstomultiplevehicles/highmobility2. Accesstosmallersetsofdata/partialviewof

thedata[22]3. Embeddeddevices/highercomputational

power4. Wireless/wiredcommunication

ThedatastreamingparadigmanditsuseinFogarchitectures

1- Distributeddeployment– wheretoplaceagivenoperator?

40

1. Globalviewofthedata(whenpossible[22])2. Accesstoaggregateddata3. Highcomputationalpower

ThedatastreamingparadigmanditsuseinFogarchitectures

M

2- Paralleldeployment– howdoweparallelize theanalysis?

41

Ifaparalleloperator(e.g.,M)feedsmultipledownstreamparalleloperators(e.g.,A),howdoweroutetuples? [23,24]

M

M A

A

ThedatastreamingparadigmanditsuseinFogarchitectures

2- Paralleldeployment– howdoweparallelize theanalysis?

42

M

M A

A

A 8:00 55.5 X1 Y1

A 8:07 34.3 X3 Y3

A 8:03 70.3 X2 Y2

A 8:00 55.5 X1 Y1

A 8:07 34.3 X3 Y3

A 8:03 70.3 X2 Y2

Whencomputinge.g.per-vehiclestatistics,weneedtomakesurethatallthetuplesreferringtothesamevehiclesaresenttothesameaggregateinstance

ThedatastreamingparadigmanditsuseinFogarchitectures

• Dependingonthestatefuloperatorsemantic:• Partitioninputstreamintobuckets• Eachbucketisprocessedby1node

• #buckets>>#nodes

43

2- Paralleldeployment– howdoweparallelize theanalysis?

ThedatastreamingparadigmanditsuseinFogarchitectures

• Dependingonthestatefuloperatorsemantic:• Partitioninputstreamintobuckets• Eachbucketisprocessedby1node

• #buckets>>#nodes

Keysdomain

Agg1 Agg2 Agg3A

D

E

B

C F

44

2- Paralleldeployment– howdoweparallelize theanalysis?

ThedatastreamingparadigmanditsuseinFogarchitectures

• Dependingonthestatefuloperatorsemantic:• Partitioninputstreamintobuckets• Eachbucketisprocessedby1node

• #buckets>>#nodes

Keysdomain

Agg1 Agg2 Agg3A

D

E

B

C F

45

2- Paralleldeployment– howdoweparallelize theanalysis?

ThedatastreamingparadigmanditsuseinFogarchitectures

2- Paralleldeployment– howdoweparallelize theanalysis?

46

Howcanweparallelizetheanalysisofe.g.anaggregatewhentheoperationsarenotperformed fordisjointpartitionsofastream(e.g.,forallvehiclesratherthanforeachvehicle?)

M

M A

A

ThedatastreamingparadigmanditsuseinFogarchitectures

47

2- Paralleldeployment– howdoweparallelize theanalysis?

ThedatastreamingparadigmanditsuseinFogarchitectures

48

M

M A

A

A 8:00 55.5 X1 Y1

A 8:07 34.3 X3 Y3

A 8:03 70.3 X2 Y2

A 8:00 55.5 X1 Y1

A 8:07 34.3 X3 Y3

A 8:03 70.3 X2 Y2

Whatiftuplewithtimestamp8:00arrivesaftertuplewithtimestamp8:07?

3– Orderinganddeterminism

ThedatastreamingparadigmanditsuseinFogarchitectures

49

3– Orderinganddeterminism

iftuplesthataggregatedtogetherarearbitrarilyinterleaved,howtoweprovidecorrectresults?

time[8:00,9:00)

8:05 8:15 8:22 8:45 9:05

Output:4

time[8:00,9:00)

8:05 8:15 8:22 8:45 9:058:55

ThedatastreamingparadigmanditsuseinFogarchitectures

50

3– Orderinganddeterminism– merge-sorting[25]&buffering [26,27]

addTuple(tuple,sourceID)allows a tuple from sourceID to be merged by ScaleGate in the resulting timestamp-sorted stream of ready tuples.

getNextReadyTuple(readerID)provides to readerID the next earliest ready tuple that has not been yet consumed by the former.

https://github.com/dcs-chalmers/ScaleGate_Java

ThedatastreamingparadigmanditsuseinFogarchitectures

4– shared-nothingvs.shared-memoryparallelism[28]

51

M

M

...

A

AJ

...

Howtotakeadvantageofmulti-corearchitectures?

Howtoboostinter-nodeparallelismandintra-nodeparallelism?

ThedatastreamingparadigmanditsuseinFogarchitectures

Shared-nothingparallelstreamjoin

ProdR

ProdS

PU1

PU2

PUN

ConsAdd tupletoPUi S

Add tupletoPUi R

Consumeresults

Pickthenextreadytuplet:1. comparetwithalltuples inoppositewindowgivenP2. addttoitswindow3. removestaletuples fromt’swindow

ChoseaPU

ChoseaPU

Takethenextreadyoutput tuple

Scalability

Highthroughput

Lowlatency

Disjointparallelism

Skewresilience

Determinism

52

Merge

4– shared-nothingvs.shared-memoryparallelism

ThedatastreamingparadigmanditsuseinFogarchitectures

ProdR

ProdS

PU1

PU2

PUN

…53

enqueue()dequeue()

ConsMerge

Shared-nothingparallelstreamjoin

4– shared-nothingvs.shared-memoryparallelism

ThedatastreamingparadigmanditsuseinFogarchitectures

ProdR

ProdS

PU1

PU2

PUN

… ConsAdd tupleSGin

Add tupleSGin

Getnextreadyoutput tuplefromSGout

Getnextreadyinput tuplefromSGin1. comparetwithalltuples inoppositewindowgivenP2. addttoitswindowinaround-robinfashion3. removestaletuples fromt’swindow

54

SGin SGout

StepsforPU

Shared-memoryparallelstreamjoin

4– shared-nothingvs.shared-memoryparallelism

ThedatastreamingparadigmanditsuseinFogarchitectures

55

t1

t2

R S

WR

t3

t4

R S

t4

t1WR

R S

t4

t2

WR

R S

t4

WR

t3

Sequentialstreamjoin:

ScaleJoinwith3PUs:

Shared-memoryparallelstreamjoin

4– shared-nothingvs.shared-memoryparallelism

ThedatastreamingparadigmanditsuseinFogarchitectures

ProdR

ProdS

PU1

PU2

PUN

… ConsAdd tupleSGin

Add tupleSGin

Getnextreadyoutput tuplefromSGout

56

SGin SGout

Scalability

Highthroughput

Lowlatency

Disjointparallelism

Skewresilience

Determinism

ProdS

ProdS

ProdR

GetnextreadyinputtuplefromSGin1. comparetwithalltuplesinoppositewindowgivenP2. addttoitswindow inaroundrobinfashion3. removestaletuplesfromt’swindow

StepsforPUi

Shared-memoryparallelstreamjoin

4– shared-nothingvs.shared-memoryparallelism

ThedatastreamingparadigmanditsuseinFogarchitectures

5– loadbalancing&statetransfer[23,29]

57

IfweshifttheprocessingofacertainsubsetoftuplesfromnodeAtonodeB,howdotransferitspreviousstate?

MA

MA

ThedatastreamingparadigmanditsuseinFogarchitectures

Statetransferchallengingforstatefuloperators

A B

time

58

5– loadbalancing&statetransfer

ThedatastreamingparadigmanditsuseinFogarchitectures

WindowRecreationProtocol[23]

A B

time

A

A

B

SendtoA

SendtoB

59

5– loadbalancing&statetransfer

+ Nocommunicationbetweennodes- Completiontimeproportionaltowindowsize

ThedatastreamingparadigmanditsuseinFogarchitectures

StateRecreationProtocol

A B

time

B

B

B

CopytoB

60

TuplesreferringtocallerA

5– loadbalancing&statetransfer

+Minimizescompletiontime- Communicationbetweennodes

ThedatastreamingparadigmanditsuseinFogarchitectures

6– elasticity[23,30]

61

How/whentoprovisionordecommissionnewresourcesdependingontheanalysis’costsfluctuations?

J

ThedatastreamingparadigmanditsuseinFogarchitectures

J J

J

Targetutilizationthreshold

Upperutilizationthreshold

Lowerutilizationthreshold

0%

100%

CPUconsumptionMemoryLatency...

………

OP OP OPOP OP

Elasticityandloadbalancingactionsonper-subclusterbasis

62

6– elasticity[23]

ThedatastreamingparadigmanditsuseinFogarchitectures

63

6– elasticity

• Howmanyresourcesshouldbeprovisioned/decommissioned?[23]

• Howtomixloadbalancingandprovisioning/decommissioning?[23]

• Howtochosewhichresourcestoprovision/decommission?[30]

• Howtopredictwhenmore/lessresourceswillbeneeded?[30]

ThedatastreamingparadigmanditsuseinFogarchitectures

Load-awareprovisioning

1 2 4 7 11 2617

64

6– elasticity[23]

ThedatastreamingparadigmanditsuseinFogarchitectures

7– faulttolerance[relatedworkin18,31,32]

65

Howtoreplaceafailednodeminimizingrecoverytime(makingittransparenttotheenduser)?

ThedatastreamingparadigmanditsuseinFogarchitectures

J

J

J

J

J

J

Activestandby

Primary

Replica

AM

A

66

Cost:

RecoveryTime:

7– faulttolerance

ThedatastreamingparadigmanditsuseinFogarchitectures

Passivestandby

Primary

Replica

AM

APeriodic

checkpoints

67

Cost:

RecoveryTime:

7– faulttolerance

ThedatastreamingparadigmanditsuseinFogarchitectures

Passivestandby

Primary

AM

Periodiccheckpoints

68

Cost:

RecoveryTime:

Disk

7– faulttolerance

ThedatastreamingparadigmanditsuseinFogarchitectures

UpstreamBackup

AM Buffer

69

Cost:

RecoveryTime:

7– faulttolerance

ThedatastreamingparadigmanditsuseinFogarchitectures

FaulttoleranceandelasticitycanalsobeintegratedtogetherinsolutionthatexternalizethemanagementofstatefromtheSPE[32]

70

7– faulttolerance

ThedatastreamingparadigmanditsuseinFogarchitectures

8 – datasharing(differentialprivacy)[2,33,34,35]

71

Howtopreventprivacyleaks?

Supposeweareinterestedinpublishingvehicles’averagespeedoverawindowofonehour...

WecouldaggregatebyRSU!

ThedatastreamingparadigmanditsuseinFogarchitectures

8 – datasharing(differentialprivacy)

72

Waitamoment!what ifasinglevehicleisconnectedtoacertainRSU?

Whetheracertainmechanismpreservesornottheprivacyoftheunderlyingdatadependsontheknowledgeoftheadversary

Differentialprivacyassumestheworstcasescenario!

ThedatastreamingparadigmanditsuseinFogarchitectures

73

8 – datasharing(differentialprivacy)

time[8:00,9:00)

22.3Km/h 10Km/h ? 15Km/h

avgspeed:30Km/h

withoutdifferentialprivacy,ifknowsthespeedof,and

andtheavgspeed,thenitcanfindoutthespeedof

ThedatastreamingparadigmanditsuseinFogarchitectures

74

8 – datasharing(differentialprivacy)

time[8:00,9:00)

22.3Km/h 10Km/h ? 15Km/h

avgspeed:30Km/h

withdifferentialprivacy,ifknowsthespeedof,and,itdoes

gainextraknowledgefromtheavgspeedinordertofindthespeedof

+noise(calibratedfromprobabilisticdistribution) sothatwhetherparticipatesornot,theresultswillbenearlythesame

ThedatastreamingparadigmanditsuseinFogarchitectures

75

8 – datasharing(differentialprivacy)

ThedatastreamingparadigmanditsuseinFogarchitectures

Agenda

• Introduction:Fogarchitectures• Datastreamingbasics• Trade-offsandchallengesinFogstreaminganalysis• Conclusions• Bibliography

76ThedatastreamingparadigmanditsuseinFogarchitectures

Introduction:Fogarchitectures

77

• wheredoweprocessdata?• howdoweprocessdata?• howdoweenforcesecurityandprivacy?

...cannotreallyanswertothesequestionsindependently...

ThedatastreamingparadigmanditsuseinFogarchitectures

Fogarchitectures,applicationstrade-offsandchallenges

78

Sensor/EdgeDevices FogDevices CloudDevices

Cost + ++ +++

Computationalpower + ++ +++

Inter-nodeheterogeneity +++ ++ +

Intra-node heterogeneity + + +++

Size +++ ++ +

Communication wireless wireless/wired wired

Volumeofdata millionsofsmallstreams thousands ofmediumaggregatedstreams manylargestreams

Evolutionovertime +++ ++ +

ThedatastreamingparadigmanditsuseinFogarchitectures

Agenda

• Introduction:Fogarchitectures• Datastreamingbasics• Trade-offsandchallengesinFogstreaminganalysis• Conclusions• Bibliography

79ThedatastreamingparadigmanditsuseinFogarchitectures

Bibliography1. Zhou,Jiazhen,RoseQingyang Hu,andYiQian."Scalabledistributedcommunicationarchitectures tosupportadvanced

metering infrastructureinsmartgrid."IEEETransactionsonParallelandDistributedSystems23.9(2012):1632-1642.2. Gulisano,Vincenzo,etal."BES:DifferentiallyPrivateandDistributedEventAggregationinAdvancedMeteringInfrastructures."

Proceedingsofthe2ndACMInternationalWorkshoponCyber-PhysicalSystemSecurity.ACM,2016.3. Gulisano,Vincenzo,MagnusAlmgren,andMarinaPapatriantafilou."Onlineandscalabledatavalidation inadvancedmetering

infrastructures."IEEEPESInnovativeSmartGridTechnologies,Europe.IEEE,2014.4. Gulisano,Vincenzo,MagnusAlmgren,andMarinaPapatriantafilou."METIS:atwo-tierintrusiondetectionsystemforadvanced

metering infrastructures."InternationalConferenceonSecurityandPrivacyinCommunicationSystems.Springer InternationalPublishing,2014.

5. Yousefi,Saleh,MahmoudSiadat Mousavi,andMahmoodFathy."Vehicularadhocnetworks(VANETs):challengesandperspectives."20066thInternationalConferenceonITSTelecommunications. IEEE,2006.

6. ElZarki,Magda,etal."Securityissuesinafuturevehicularnetwork."EuropeanWireless. Vol.2.2002.7. Georgiadis,Giorgos,andMarinaPapatriantafilou."Dealing withstoragewithoutforecastsinsmartgrids:Problem

transformationandonlineschedulingalgorithm."Proceedingsofthe29thAnnualACMSymposiumonAppliedComputing.ACM,2014.

8. Fu,Zhang,etal."Onlinetemporal-spatialanalysisfordetectionofcritical eventsinCyber-PhysicalSystems."BigData(BigData),2014IEEEInternationalConferenceon.IEEE,2014.

ThedatastreamingparadigmanditsuseinFogarchitectures 80

Bibliography9. Arasu,Arvind, etal."Linearroad:astreamdatamanagementbenchmark."ProceedingsoftheThirtieth

international conferenceonVerylargedatabases-Volume30.VLDBEndowment,2004.10. Lv,Yisheng,etal."Trafficflowpredictionwithbigdata:adeeplearningapproach."IEEETransactionson

IntelligentTransportationSystems16.2(2015):865-873.11. Grochocki,David,etal."AMIthreats,intrusion detectionrequirementsanddeployment

recommendations."SmartGridCommunications (SmartGridComm), 2012IEEEThirdInternationalConferenceon.IEEE,2012.

12. Molina-Markham,Andrés,etal."Privatememoirsofasmartmeter."Proceedingsofthe2ndACMworkshoponembedded sensingsystemsforenergy-efficiencyinbuilding. ACM,2010.

13. Gulisano,Vincenzo,etal."Streamcloud:Alargescaledatastreamingsystem."DistributedComputingSystems(ICDCS),2010IEEE30thInternationalConferenceon.IEEE,2010.

14. Stonebraker,Michael,Uǧur Çetintemel,andStanZdonik."The8requirementsofreal-timestreamprocessing."ACMSIGMODRecord34.4(2005):42-47.

15. Bonomi,Flavio,etal."Fogcomputing anditsroleintheinternetofthings."Proceedingsof thefirsteditionoftheMCCworkshoponMobilecloudcomputing.ACM,2012.

16. Himmelsbach,Michael,etal."LIDAR-based3Dobjectperception."Proceedingsof1stinternationalworkshoponcognition fortechnicalsystems.Vol.1.2008.

ThedatastreamingparadigmanditsuseinFogarchitectures 81

Bibliography17. Geiger,Andreas,etal."Visionmeetsrobotics:TheKITTIdataset."TheInternationalJournalofRoboticsResearch(2013):

0278364913491297.18. Gulisano,VincenzoMassimiliano. StreamCloud:AnElasticParallel-DistributedStreamProcessingEngine.Diss.Informatica,

2012.19. Cardellini, Valeria,etal."Optimaloperatorplacement fordistributedstreamprocessingapplications."Proceedingsofthe10th

ACMInternationalConferenceonDistributedandEvent-basedSystems.ACM,2016.20. Costache,Stefania,etal."UnderstandingtheData-ProcessingChallenges inIntelligentVehicularSystems."Proceedingsofthe

2016IEEEIntelligentVehiclesSymposium(IV16).21. Cormode,Graham."Thecontinuousdistributedmonitoringmodel."ACMSIGMODRecord42.1(2013):5-14.22. Giatrakos,Nikos,AntoniosDeligiannakis, andMinosGarofalakis."ScalableApproximateQueryTrackingoverHighlyDistributed

DataStreams."Proceedingsofthe2016InternationalConferenceonManagementofData.ACM,2016.23. Gulisano,Vincenzo,etal."Streamcloud:Anelasticandscalabledatastreamingsystem."IEEETransactionsonParalleland

DistributedSystems23.12(2012):2351-2365.24. Shah,Mehul A.,etal."Flux:Anadaptivepartitioningoperatorforcontinuousquerysystems."DataEngineering,2003.

Proceedings.19thInternationalConferenceon.IEEE,2003.

ThedatastreamingparadigmanditsuseinFogarchitectures 82

Bibliography25. Cederman,Daniel,etal."Briefannouncement:concurrentdatastructuresforefficientstreamingaggregation."Proceedingsof

the26thACMsymposiumonParallelism inalgorithmsandarchitectures.ACM,2014.26. Ji,Yuanzhen,etal."Quality-drivenprocessingofslidingwindowaggregatesoverout-of-orderdatastreams."Proceedingsofthe

9thACMInternationalConferenceonDistributedEvent-BasedSystems.ACM,2015.27. Ji,Yuanzhen,etal."Quality-drivendisorderhandlingforconcurrentwindowedstreamquerieswithsharedoperators."

Proceedingsofthe10thACMInternationalConferenceonDistributedandEvent-basedSystems.ACM,2016.28. Gulisano,Vincenzo,etal."Scalejoin:Adeterministic,disjoint-parallelandskew-resilient streamjoin."BigData(BigData),2015

IEEEInternationalConferenceon.IEEE,2015.29. Ottenwälder,Beate,etal."MigCEP:operatormigrationformobilitydrivendistributedcomplexeventprocessing."Proceedings

ofthe7thACMinternational conferenceonDistributedevent-basedsystems.ACM,2013.30. DeMatteis,Tiziano,andGabrieleMencagli. "Keepcalmandreactwithforesight:strategiesforlow-latencyandenergy-efficient

elasticdatastreamprocessing."Proceedingsofthe21stACMSIGPLANSymposiumonPrinciplesandPracticeofParallelProgramming.ACM,2016.

31. Balazinska,Magdalena,etal."Fault-tolerance intheBorealisdistributedstreamprocessingsystem."ACMTransactionsonDatabaseSystems(TODS)33.1(2008):3.

32. CastroFernandez,Raul,etal."Integratingscaleoutandfaulttolerance instreamprocessingusingoperatorstatemanagement."Proceedingsofthe2013ACMSIGMODinternational conferenceonManagementofdata.ACM,2013.

ThedatastreamingparadigmanditsuseinFogarchitectures 83

Bibliography

33. Dwork,Cynthia."Differentialprivacy:Asurveyofresults."InternationalConferenceonTheoryandApplicationsofModelsofComputation.SpringerBerlinHeidelberg,2008.

34. Dwork,Cynthia,etal."Differentialprivacyundercontinualobservation."Proceedingsoftheforty-secondACMsymposiumonTheoryofcomputing.ACM,2010.

35. Kargl,Frank,ArikFriedman,andRoksana Boreli."Differentialprivacyinintelligenttransportationsystems."ProceedingsofthesixthACMconferenceonSecurityandprivacyinwirelessandmobilenetworks.ACM,2013.

ThedatastreamingparadigmanditsuseinFogarchitectures 84

Thankyou!Questions?

85ThedatastreamingparadigmanditsuseinFogarchitectures