science data and the ndn paradigm · 2016. 9. 13. · program-mable network extreme data science...

ScienceDataandtheNDNparadigmInderMongaCTO,ESnetDivisionDeputyofTechnology,Scien?ficNetworkingDivisionLawrenceBerkeleyNa?onalLab NDNComm2015

Experimental and observational science deals with big and small

instruments, and a lot of data!

2

●  DatavolumesareincreasingfasterthanMoore’sLaw

●  Newalgorithmsandmethodsforanalyzingdata

●  Infeasibletoputasupercompu>ngcenterateveryexperimentalfacility

Compu?ngSciencesArea

Alltoocommonprocessofdiscovery

3

ESnet

NewMath

Real->meanalysis

HighperformanceSoFware

Novelcompute/dataplaHorms

Datamgmt.andsharing

Program-mablenetwork

Extreme Data Science Facility

(XDSF)

MS-DESI

ALS

LHC

JGI

APS

LCLS

Other data-producing sources

-4-

‘Superfacility’Vision:Anetworkofconnectedfacili>es,soFwareandexper>setoenablenewmodesofdiscovery

ESnetisadedicatedmissionnetworkengineeredtoaccelerateabroadrangeofscienceoutcomes.

Wedothisbyofferinguniquecapabili?es,andop?mizingthenetworkfordataacquisi?on,data

placement,datasharing,datamobility.

ESnetisdesignedfordifferentgoalsthan

generalInternet.

Feb-94 Apr-98 Jun-02 Aug-06 Oct-10 Dec-14

1017

1010

1011

1012

1013

1014

1015

1016

Month

Tota

l

Bytes of Science Data Transferred Each Monthby the Energy Sciences Network

August2015:29.13PB

Lotsofdatatomovearound

Feb-94 Apr-98 Jun-02 Aug-06 Oct-10 Dec-14

3 × 1016

0

5 × 1015

1 × 1016

1.5 × 1016

2 × 1016

2.5 × 1016

Month

Tota

l

Bytes of Science Data Transferred Each Month

by the Energy Sciences Network

August2015:29.13PB

Lotsofdatatomovearound(contd.)

High-levelobjec>vesforscien>ficdata:alignmentwithNDNapproach

10/25/159

•  Radicallysimplifyhowscien?ficusersmanage,moveandmanipulatelarge,distributed,sciencedatarepositories,butwithhigh-throughputend2end

•  Abstractthestorageandnetworkcapabilityandloca?ondependencefromtheuser-datainterac?on

•  Enabletheabilityforuserstospecifyandretrievepor?onsofdatatheworkflowneeds

•  Createasecure,scalableframeworkbasedonintegrateddatamanagementandnetworktransport

UseCase#1

10/25/1510

ResearchersfromBerkeleyLabandSLACconductedproteincrystallographyexperimentsatLCLStoinves>gatephotoexcited

statesofPSII,withnear-real->mecomputa>onalanalysisatNERSC.

“Takingsnapshotsofphotosynthe?cwateroxida?onusingfemtosecondX-raydiffrac?onandspectroscopy,”NatureCommunica.ons5,4371(9July2014)

50TBmovedanight

UseCase#2:LHCONEdata–mul>plereplicas,globalreach

UseCase#3:Interna>onalClimateData

10/25/1512

Percep>onoflimita>onsofNDNmo>va>ngresearchques>ons

1.  IfIammoving50TBofdatathroughasinglepath,fromanexperimenttoastoragefacility,IreallydonotwanttocacheitateveryintermediateNDNnode

–  Whatistherightstrategyforalloca?ngdiskresourcestocaching?Whatifonedatatransferconsumesallcacheresourcesorthereisnotenoughspace?

2.  Whatistheperformanceoftheend-to-enddatatransfer?HowcanIgetlineratethroughput?

3.  HowdoIleveragetheknowledgeofnetworkcapabilityinchoosingthetransferpath?HowdoIbuildintheknowledgeofunderlayintotheNDNoverlay?

4.  HowdoIleveragenetworkprogrammabilitytodotheabove?

5.  Andmanyotherques?ons….

10/25/1513

Whereareweat?

•  Collabora?onwithChristosandColoradoState–high-poweredNDNdevicesbetweenthreerepresenta?veclimatesitesasatestbed–  Susmitworkingonansweringsomeofthehigh-levelobjec?vesasdescribed

•  HEPandASCRinterestinNDNfromaresearchperspec?ve–paperearlierthisyear@CHEP,andPhilwilltalkaboutnext-stepsrightaher

•  Interestinexpandingafedera?onofhigh-poweredNDNdeviceswiththerightstrategyforcachinganddatamanagement

•  CombiningNDNwithSDN–wehaveanext-genSDNtestbedacrossUSandEurope–canwecombinethattoprovidetherightprimi?vesforhigh-performanceNDN?–  Letsdoitera?veexperimenta?onandimprovement!!!!!!!

10/25/1514

ALBQ

AMST

ANL

AOFA

ATLA

BNL

BOIS

BOST

CERN

CHICDENV

ELPA

FNAL

HOUS

KANS

LANL

LBL LLNL

LOND

NASHNERSC

NEWY

ORNL

PNNLPNWG

SACR

SAND

SLAC

STAR

SUNN

WASH

ESnet PE Router

(2+)x10GE

(n)x10GE

Testbed Host

Deployed SDN Testbed node locations Deployed SDN Testbed connectivity overlay (using OSCARS circuits)

ESnetSDNTestbed

AMST

CERN

AOFA

WASH

STAR

ATLA

DENV

LBL

August2015 iDiscovery2020InderMonga15

StatusUpdate:•  Testbeddeployedatallloca?ons•  QoSsupportverified,pressrelease

nextweek•  ENOSdemoonTestbed@SC

Thankyou!

•  Pleasefeelfreetoemailmewithques?ons,commentsorarrowsat

imongaatesdotnet

10/25/1516

10/25/1517