university of southern california · data social sci. power engr. facility mgmt. univ. staff extern...
TRANSCRIPT
USC Center for Energy Informatics cei.usc.edu
Consumers
Information Technology
Energy Technology
Microsoft Cloud Futures Workshop 3
USC Center for Energy Informatics cei.usc.edu
INFR
AST
RU
CTU
RE
SOC
IAL
CYBER SPACE PHYSICAL SPACE
Compute & Data Cyber
Infrastructure
Physical Microgrid
Infrastructure
Online Social Networks
Physical Social Interactions
H H
H
H
H H
H
Sensor
Actuator
Microsoft Cloud Futures Workshop 4
USC Center for Energy Informatics cei.usc.edu
“The Smart Grid Explained, The Hype and The Promise”, WESCO, FMEA/FMPA Conference 2009
Microsoft Cloud Futures Workshop 5
USC Center for Energy Informatics cei.usc.edu
US EIA’s Annual Energy Outlook 2011
Source: GDF Suez
Microsoft Cloud Futures Workshop 7
USC Center for Energy Informatics cei.usc.edu
Software Platform
Energy System
Consumer Behavior
Microsoft Cloud Futures Workshop 8
USC Center for Energy Informatics cei.usc.edu
Pillcrow Graph Analytics
Cryptonite
Learn from the Microgrid. Scale to the Mega-city grid.
Security & Privacy
Scalable Machine Learning
Consumer Behavior
Model
Building Curtailment
Model
Demand Forecasting DR Policy Engine
Trigger
Select
Initiate
Campus
Complex Event Pattern
Detection Observe & Adapt
Observe & Ingest
Information Repository
Mine & Model
Track
SCEPter CEP Engine
OpenPLANET MapReduce
CRAFT Causal Models
Floe Dataflow Engine
Stream Provenance
Portal & Apps
USC Center for Energy Informatics cei.usc.edu
Data Social Sci.
Power Engr.
Facility Mgmt.
Univ. Staff
Extern
Cons. Survey O R* - - -
Staff Schedule R* R* - O -
Consumption R R O - -
Weather R R O R W Campus Power Forecast
R W O R -
Shared Data Repository
Social Scientist
Power Engineering Researcher
Facility Managemen
t Student
Staff
Building Sensors
13
USC Center for Energy Informatics cei.usc.edu
17c96af0-02a2-4ed6-8b45-8b436e6ea79d
Filename: usc-upc/eeb/kwh-2011- 09-15.csv
... data...
fa9457ef-66bc-4bc4-88a5-472b5c2ac75b
e3256660-8e27-4d0c-aca0-4ce33a104c06
Repository Service
OWNER
WR
ITER R
EAD
ER U
NA
UTH
OR
IZED
Microsoft Cloud Futures Workshop 16
USC Center for Energy Informatics cei.usc.edu
F = encrypt(D, Ksymencr)
SF = sign(F, Kpri
sign)
M = <UUID, SUUID, Kpubsign, Uowner>
SM = sign(M, Kpri
owner)
SB =
encrypt(L, Ksymencr)
broad-encrypt(Ksymencr, Kshared
r-encr)
broad-encrypt(Kprisign, Kshared
w-encr)
Uowner IV-StrongBox
sign(SB, Kpriowner)
File Handle Init. Vector
UUID-1 IV-1
UUID-2 IV-2
…
L =
Data File
Strong Box
Designing a Secure Storage Repository for Sharing Scientific Datasets using Public
Clouds, A. Kumbhare, Y. Simmhan & V. Prasanna , DataCloud, 2011
Microsoft Cloud Futures Workshop 18
USC Center for Energy Informatics cei.usc.edu
Cryptonite Data Repository Service
File Manager
Audit Manager
Azure Web Role VM Azure BLOB Store
Azure Table Store Cryptonite Audit Table
WINDOWS AZURE CLOUD PLATFORM DESKTOP CLIENT
Cryptonite Client Library
File Manager
Audit Manager
1a 1b
1c
StrongBox Processor
Crypt Layer
2a
DFile Processor
3 4 Upload Processor
Crypt Layer
Download Processor
5 6a, 12a
8,14
7a, 13a 9a,15a
7b, 13b
9b, 15b
10 Cryptonite DFiles
Cryptonite Strongbox Files
Local Disk Plain Text File
11
2b
6b, 12b
Trusted PKI Server
Operations • GET • PUT • POST
Microsoft Cloud Futures Workshop 19
USC Center for Energy Informatics cei.usc.edu
Owner
Cryptonite
FM AM
Request StrongBox
a)Encrypt & Sign File b)Generate UUID c) Add and Sign Security
Metadata Encrypted file with security
metadata
Verify Signature and Store in CSS
Add Audit Entry
Signed Confirmation Containing File Hash
Verify Owner and Store in CSS
Add Audit Entry
Signed Confirmation Containing StrongBox Hash
Upload StrongBox
a)Update StrongBox b)BEr+w(Ke), BEw(Ks) c) Encrypt StrongBox
Microsoft Cloud Futures Workshop 20
USC Center for Energy Informatics cei.usc.edu
E S T F R V U
E T
F R V U 3 2 1 S T
U 3 2 1
H H
3 2 1
E T
F R V U 3 2 1 S T
U 3 2 1
H H
3
2
1
U U
R U S T
U
3
2
1
H H
3
2
1
U U
F E
3
2
1
E E
3 2 1
T
3
2
1
T T
V 3 2 1 H
Desktop Client Service BLOB Store Internet
(a)
(b)
(c)
(d)
Cryptonite: A Secure and Performant Data Repository on Public Clouds, A. Kumbhare, Y. Simmhan & V. Prasanna , IEEE CLOUD, 2012
Microsoft Cloud Futures Workshop 21
USC Center for Energy Informatics cei.usc.edu
Brandes, U.: A faster algorithm for betweenness centrality. J. Mathematical Sociology, 25(2), 163–177 (2001)
Microsoft Cloud Futures Workshop 28
USC Center for Energy Informatics cei.usc.edu
Performance Analysis of Vertex-centric Graph Algorithms on the Azure Cloud Platform, M. Redekopp, Y. Simmhan & V. Prasanna, ParGraph, 2011
Web
Instance Manager
Instance
Workers
Results (Blob Storage)
Requests
Notification
Task Queue
Result Notification Queue
Graphs (Blob Storage)
Microsoft Cloud Futures Workshop 29
USC Center for Energy Informatics cei.usc.edu
0
2
4
6
8
10
12
14
16
4 workers 8 workers 16 workers
Spee
du
p R
elat
ive
to 1
VM
Number of BC Worker Instances
Speedup Relative to 1-VM
10,000 vertices
100,000 vertices
0
1
2
3
4
5
6
7
8
1 worker 4 workers 8 workers 16 workersSp
eed
up
Rel
ativ
e to
1 L
oca
l Co
reNumber of BC Worker Instances
Speedup Relative to 1 Local Core
1000 vertices
10,000 vertices
100,000 vertices
For large graphs performance is
scalable over number of workers For large graphs and 16 workers we get about 8x
speedup compared to a single local core
Microsoft Cloud Futures Workshop 30
USC Center for Energy Informatics cei.usc.edu
0%
20%
40%
60%
80%
100%
-12% -8% -4% 0% 4% 8% 12%
Task Completion Time Normalized to the Average over all Tasks
Task Size 1000
Task Size 500
Task Size 100
For 105 vertex graph, all tasks finish with
in 12% variation from the average
Set timeout to average task time+12% Analyze performance penalties for different MTBF
More tasks per worker (smaller task size) helps to mitigate performance impact of failures
Microsoft Cloud Futures Workshop 31
USC Center for Energy Informatics cei.usc.edu
Pregel: a system for large-scale graph processing, G. Malewicz, et al., SIGMOD 2010
Superstep 2
Superstep 1
Superstep 3
…
32
USC Center for Energy Informatics cei.usc.edu
Socket
Conne-
ctions
Web Instance
Manager Instance
Workers
Results (Blob Storage)
Requests
Notification
Step Queue
Barrier Queue
Graphs (Blob Storage)
1
1 1
1
Superstep 2
Workers Superstep 1
Superstep 3
…
Azure
Desktop
Pregel.NET Worker Role Implementation on Azure
Microsoft Cloud Futures Workshop 33
USC Center for Energy Informatics cei.usc.edu
GetThreads (1 per remote worker connection)
ComputeThreads
Local Vertices Message Queues for next Superstep
Local Vertices Message Queues for current Superstep
Put Buffers (Remote Messages)
PutThreads (1 per remote worker connection)
Microsoft Cloud Futures Workshop 34
USC Center for Energy Informatics cei.usc.edu
0
200000
400000
600000
800000
1000000
1200000
1400000
1600000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19N
um
be
r o
f Me
ssag
es
Ge
ne
rate
d p
er
BFS
SuperStep
Microsoft Cloud Futures Workshop 35
USC Center for Energy Informatics cei.usc.edu
0
20000
40000
60000
80000
100000
120000
140000
1 2 3 4 5 6 7 8 9 10 11 12 13
0
20000
40000
60000
80000
100000
120000
140000
160000
180000
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Message traffic from 4 swaths of BFS launched at an interval of 2 supersteps
Cumulative message traffic for the 4 swaths
Microsoft Cloud Futures Workshop 36
USC Center for Energy Informatics cei.usc.edu
This material is based upon work supported by the National Science Foundation and Microsoft under Award Number CCF-1048311 of the Computing in the Cloud initiative, and by the Department of Energy under Award Number DEOE0000192.
The views and opinions expressed herein do not necessarily state or reflect those of the US Government or any agency thereof.
Microsoft Cloud Futures Workshop 40