![Page 1: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/1.jpg)
Cees de Laat!GLIF.is founding member!
OFC 2009 – Grid vs Cloud Computing and Why This Should Concern the Optical Networking Community
Challenges in Enabling Grid Computing
over Optical Networks
![Page 2: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/2.jpg)
100000 flops/byte!
![Page 3: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/3.jpg)
Telescopes
Input nodes
Correlator nodes
Output node
..... To equal the hardware
correlator we need:
16 streams of 1Gbps
16 * 1Gbps of data
2 Tflops CPU power
2 TFlop / 16 Gbps =
1000 flops/byte
THIS IS A DATA FLOW
PROBLEM !!!
SCARIe: a research project to create a Software Correlator for e-VLBI. VLBI Correlation: signal processing technique to get high precision image from
spatially distributed radio-telescope.
The SCARIe project
![Page 4: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/4.jpg)
The “Dead Cat” demo" SC2004 & iGrid2005!
SC2004,
Pittsburgh,
Nov. 6 to 12, 2004
iGrid2005,
San Diego,
sept. 2005
Produced by:
Michael Scarpa
Robert Belleman
Peter Sloot
Many thanks to:
AMC
SARA
GigaPort
UvA/AIR
Silicon Graphics,
Inc.
Zoölogisch Museum
![Page 5: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/5.jpg)
US and International OptIPortal Sites!
NCMIR SIO
UIC
USGS EDC
TAMU UCI
SARA KISTI
NCSA &
TRECC
CALIT2
AIST RINCON & Nortel
![Page 6: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/6.jpg)
IJKDIJK!
![Page 7: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/7.jpg)
Sensor grid: instrument the dikes!
30000 sensors (microphones) to cover all Dutch dikes!
First controlled breach occurred on sept 27th ‘08:!
![Page 8: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/8.jpg)
e-F
oo
d!
e-B
ioS
cie
nce!
e-B
iob
an
kin
g!
e-C
OA
ST!
e-E
co
log
y!
e-D
ata
-
inte
nsiv
e
scie
nc
es!
.....!
Virtual Laboratory!
generic e-Science services!
High Performance & Distributed Computing!
Web & Grid services!
![Page 9: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/9.jpg)
GLIF 2008! Visualization courtesy of Bob Patterson, NCSA Data collection by Maxine Brown.
![Page 10: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/10.jpg)
![Page 11: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/11.jpg)
Management
Visualisation
Mining
Web2.0
Media
Backup
Security
NetherLight!Meta
CineGrid
Medical
TV
Gaming
Conference
RemoteControl
Clouds
Distributed Simulations
Predictions
EventProcessing StreamProcessing
Workflow
DataExploration
![Page 12: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/12.jpg)
Who (just to name a few) working Interfaces - similar
solutions addressing the previous challenges
•! CANARIE - UCLP
•! ESnet = OSCARS
•! G-LAMBDA - GNS-WSI
•! HPDM - VLAN based lightpaths
•! NORTEL - DRAC
•! UvA - Token based service, NDL, etc
•! Phosphorous - G2MPLS, UCLP2, NRPS - > NSP
•! GN2 - JRA3 - AutoBahn - IDM
•! I2 - DRAGON/ HOPI - DCN
•! FermiLab - Lambda Station
•! DoE - LambdaStation, TeraPaths
Slide: Gigi Karmous Edwards!
![Page 13: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/13.jpg)
•! Harmony architecture (I) !
![Page 14: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/14.jpg)
![Page 15: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/15.jpg)
•!15!
Grid-aware GMPLS (G2MPLS) for Grid Network Services
!! Two models for the layering between Grid and Network resources
G2MPLS overlay model
different scope with respect to the IETF GMPLS Overlay & Peer
G2MPLS integrated model
![Page 16: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/16.jpg)
My view!
GLIF!
GN2/3!
Federica !
Phosphorus!
Onelab!
Planetlab!
FIRE!
etc.!
OSI!
7!
6!
5!
4!
3!
2!
1!
0!
•! needs
repeatable
experiments!
•! needs QoS &
lightpaths!
•! needs capacity
and capability!
•! needs
infrastructure
descriptions!
![Page 17: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/17.jpg)
TeraThinking!
•! What constitutes a Tb/s network?!
•! CALIT2 has 8000 Gigabit drops ?->? Terabit Lan?!
•! look at 80 core Intel processor!
–! cut it in two, left and right communicate 8 TB/s!
•! think back to teraflop computing!!
–! MPI turns a room full of pc’s in a teraflop machine!
•! massive parallel channels in hosts, NIC’s!
•! TeraApps programming model supported by!
–! TFlops ! !-> !MPI / Globus!
–! TBytes ! !-> !OGSA/DAIS!
–! TPixels ! !-> !SAGE!
–! TSensors !-> !LOFAR, LHC, LOOKING, CineGrid, ...!
–! Tbit/s ! !-> !?!
ref Larry Smarr & CdL!
![Page 18: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/18.jpg)
The Problem!
A
G
F E
D
C
B
H
I want HC and AB!
Success depends on the order!
Wouldn’t it be nice if I could request [HC, AB, ...]!
?!
![Page 19: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/19.jpg)
Network Description Language
Object Subject Predicate
•! From semantic Web / Resource Description Framework.
•! The RDF uses XML as an interchange syntax.
•! Data is described by triplets:
Location! Device! Interface! Link!
name! description! locatedAt! hasInterface!
connectedTo! capacity! encodingType! encodingLabel!
Object
Subject Subject
Object
Subject
Object
Subject
Object
Subject
![Page 20: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/20.jpg)
Network Description Language
Choice of RDF instead of XML syntax
Grounded modeling based on G0805 description:
Article: F. Dijkstra, B. Andree, K. Koymans, J. van der Ham, P. Grosso, C. de Laat, "A Multi-Layer Network
Model Based on ITU-T G.805"
![Page 21: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/21.jpg)
NDL + PROLOG!
•!Reason about
graphs!
•!Find sub-graphs
that comply with
rules !
Research Questions:!
•!order of requests!
•!complex requests!
•!Usable leftovers!
![Page 22: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/22.jpg)
139.63.145.0
139.63.145.1
139.63.145.15
139.63.145.16139.63.145.17
139.63.145.18
139.63.145.2
139.63.145.3
139.63.145.31
139.63.145.32
139.63.145.33
139.63.145.34
139.63.145.38
139.63.145.40
139.63.145.41
139.63.145.42139.63.145.43
139.63.145.44
139.63.145.45
139.63.145.46
139.63.145.49
139.63.145.50
139.63.145.51
139.63.145.52
139.63.145.63
139.63.145.64
139.63.145.65
139.63.145.66
139.63.145.68
139.63.145.69139.63.145.70139.63.145.71
139.63.145.72
139.63.145.73139.63.145.74
139.63.145.75
139.63.145.79
139.63.145.81
139.63.145.82
139.63.145.83
139.63.145.84
139.63.145.85
139.63.145.86
139.63.145.87
139.63.145.88
139.63.145.94 192.168.0.1 192.168.0.2
192.168.0.3
192.168.0.4
192.168.0.5
192.168.0.6
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
Visualisation
Initialization and BFS discovery of NEs
95.9
94.5
95.8
96.
95.9
99.9
99.
100.
100.
94.599.9
99.9
100.
95.899.
96.
99.9
100.
100.
99.8
100.
100.
98.9
99.8
100.
100.
100.
100.
100.
98.9
101.
100.
101.100.
100.
100.
139.63.145.94192.168.0.1192.168.0.2
192.168.0.3
192.168.0.4
192.168.0.5
192.168.0.6
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4
192.168.2.1
192.168.2.2
192.168.2.3
192.168.2.4
Network flows using real-time bandwidth measurements
nodePath = ConvertIndicesToNodes[
ShortestPath[ g,
Node2Index[nids,"192.168.3.4"],
Node2Index[nids,"139.63.77.49"]],
nids];
Print["Path: ", nodePath];
If[NetworkTokenTransaction[nodePath, "green"]==True,
Print["Committed"], Print["Transaction failed"]];
Path:
{192.168.3.4,192.168.3.1,139.63.77.30,139.63.77.49}
Committed
Transaction on shortest path with tokens
Topology matters can be dealt with algorithmically
Results can be persisted using a transaction service built in UPVN
Needs["WebServices`"]
<<DiscreteMath`Combinatorica` <<DiscreteMath`GraphPlot`
InitNetworkTopologyService["edge.ict.tno.nl"]
Available methods:
{DiscoverNetworkElements,GetLinkBandwidth,GetAllIpLinks,Remote, NetworkTokenTransaction}
Global`upvnverbose = True;
AbsoluteTiming[nes = BFSDiscover["139.63.145.94"];][[1]]
AbsoluteTiming[result = BFSDiscoverLinks["139.63.145.94", nes];][[1]]
Getting neigbours of: 139.63.145.94
Internal links: {192.168.0.1, 139.63.145.94} (...)
Getting neigbours of:192.168.2.3
Internal links: {192.168.2.3}
Mathematica enables advanced graph queries, visualizations and real-
time network manipulations on UPVNs"
ref: Robert J. Meijer, Rudolf J. Strijkers, Leon Gommans, Cees de Laat, User Programmable Virtualiized Networks, accepted for publication to the IEEE e-Science 2006 conference Amsterdam.
![Page 23: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/23.jpg)
Interactive programmable networks
![Page 24: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/24.jpg)
Multi Layer Service Architecture!
Network
layers
Application
layers Use Interface
Control Interface
(protocols API’s)
Network
Service
Application
5b of 6!
![Page 25: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/25.jpg)
n.a.v. interview met Kees Neggers (SURFnet) & Cees de Laat (UvA) "
•!BSIK projects
GigaPort &!
•!VL-e / e-
Science!
cookreport.com!
![Page 26: OFC 2009 – Grid vs Cloud Computing and Why This Should … - Challenges in Enabling Grid... · 16 streams of 1Gbps 16 * 1Gbps of data 2 Tflops CPU power 2 TFlop / 16 Gbps = 1000](https://reader034.vdocuments.us/reader034/viewer/2022052004/6017becec670c579266186cf/html5/thumbnails/26.jpg)
Questions ?!
Thanks: Paola Grosso & Jeroen vd Ham & Freek
Dijkstra & team for several of the slides.!
A Declarative Approach to Multi-Layer Path Finding Based on Semantic Network Descriptions.!
http://delaat.net:/~delaat/papers/declarative_path_finding.pdf!