managing scientific data with ndn · 2015-09-29 · managing scientific data with ndn chengyu fan,...

MANAGING SCIENTIFIC

DATA WITH NDN

Chengyu Fan, Susmit Shannigrahi, Steve DiBenedetto,

Catherine Olschanowsky, Christos Papadopoulos

NDNcomm 2015

Sept 28, 2015 Los Angeles, CA

Supported by NSF #13410999 and NSF#1345236

Introduction

Scientific data is often very large and complex

Climate - CMIP5: 3.5 PB, CMIP6: 350PB-3EB

Physics - Atlas: 4 PB/Year

Astronomy, bioinformatics, others…

Science infrastructure

Cutting edge hardware but often incompatible

domain software (ESGF, xrootd, etc.)

Complexity, replication, redundancy

Our Project

Build and deploy software to evaluate NDN in

scientific applications over a dedicated hardware

infrastructure

Evaluate NDN in the context of:

Application services: publishing, discovery, retrieval, access

control, load balancing, failover, caching, etc.

Network integration (OSCARS, SDN, etc.)

Metrics

Performance, reduced complexity, ease of deployment,

interoperability, reuse, efficiency, routing, security/trust, etc.

NDN Layer Structure

UDP/IP

host host

UDP/IP

NDN Layer Structure

UDP/IP

host host

UDP/IP

NDN Layer Structure

UDP/IP

router

UDP/IP

NDN Layer Structure

UDP/IPETH

router

UDP/IPETH

NDN Layer Structure

UDP/IPETH

router

UDP/IPETH

NDN Layer Structure

UDP/IPETH

router

UDP/IPETH

NDN Layer Structure

UDP/IPETH

router

UDP/IPETH

router

Methodology

Investigate the use of NDN as a common

platform for scientific data applications by:

Understanding data management challenges of

various scientific domains

Developing and evaluating prototype applications

that leverage NDN's features

Use prototypes to further drive NDN research

First Step – Build a Catalog

Create a shared resource – a distributed, synchronized

catalog of names over NDN

Provide common operations such as publishing, discovery, access control

Catalog only deals with name management, not dataset retrieval

Platform for further research and experimentation

Research questions:

Namespace construction, distributed publishing, key management, UI

design, failover, etc.

Functional services such as subsetting

Mapping of name-based routing to tunneling services (VPN, OSCARS,

Overview of Catalog Workflow

Catalog node 1

Data storage

Publisher

Catalog node 2

Consumer

Catalog node 3

Catalog node 1

Data storage

(1)Publish Dataset

Publisher

Catalog node 2

Consumer

Catalog node 3

Catalog node 1

Data storage

Publisher

Catalog node 2

Consumer

Catalog node 3

Catalog node 1

Data storage

Publisher

Catalog node 2

(2) Sync changes

Consumer

Catalog node 3

Catalog node 1

Data storage

Publisher

Catalog node 2

Consumer

Catalog node 3

Catalog node 1

Data storage

Data storage(3) Query for

Dataset names

Publisher

Catalog node 2

Consumer

Catalog node 3

Catalog node 1

Data storage

Publisher

Catalog node 2

Consumer

Catalog node 3

Catalog node 1

Data storage

Publisher

(4) Retrieve data

Catalog node 2

Consumer

Catalog node 3

Catalog node 1

Data storage

Publisher

(4) Retrieve data

Catalog node 2

Consumer

Catalog node 3

Catalog node 1

Data storage

Publisher

(4) Retrieve data

Catalog node 2

Consumer

Catalog node 3

NDN-Science Testbed

NSF CC-NIE campus infrastructure award

10G testbed (courtesy of ESnet, UCAR, and CSU Research LAN)

Currently ~50TB of CMIP5, ~70TB of HEP data

Search

Publication and Sync

Access control

Retrieval and failover

Conclusions

IP encourages common host access, not common data access

methods

Does not encourage interoperability at the application level

NDN has the potential to unify the service interface required

by scientific applications

Science testbed and prototypes to test hypothesis and drive research

and experimentation

Ready-to-try catalog, we invite you to try it with your data

Catalog is general, supports a variety of applications

Currently CMIP5 and HEP applications

UI for data search and retrieval.

Our sponsors: NSF and ESnet

Join us @

http://www.netsec.colostate.edu/mailman/listinfo/ndn-sci

Backup Slides

Current Example: xrootd

/my/file /my/file

Data Serverscmsdxrootd cmsdxrootd cmsdxrootd

Fragile, fairly complex middleware

/my/file /my/file

Data Servers

Manager(a.k.a. Redirector)

cmsdxrootd cmsdxrootd cmsdxrootd

cmsdxrootd

/my/file /my/file

Data Servers

Client

cmsdxrootd

/my/file /my/file

4: Try open() at A

Data Servers

Client

cmsdxrootd

xrootd under NDN

Significantly reduced system complexity

Better service abstraction

/my/file /my/file

xrootd under NDN

/my/file /my/file

xrootd under NDN

/my/file /my/file

Data Servers

Client

xrootd under NDN

/my/file /my/file

Data Servers

Client

? /my/file

xrootd under NDN

/my/file /my/file

Data Servers

Client

? /my/file

Data Publication

PublisherCatalog

1) Listening on /<catalog-

prefix>/publish

Data Publication

PublisherCatalog

prefix>/publish

2) Generate NDN names for

datasets/services

Data Publication

PublisherCatalog

3) Request publish

prefix>/publish

datasets/services

Data Publication

PublisherCatalog

3) Request publish

4) Fetch published name list

prefix>/publish

datasets/services

Data Publication

PublisherCatalog

3) Request publish

5) Authenticate the Data and

validate data name against trust

prefix>/publish

datasets/services

Data Publication

PublisherCatalog

3) Request publish

6) Share names with other

catalogs

5) Authenticate the Data and

validate data name against trust

prefix>/publish

datasets/services

Keys for ndn-atmos

Self-signed root key/cmip5/KEY

/cmip5/lbl/KEY /cmip5/nwsc/KEY… Site’s keys

/cmip5/lbl/<DataPublisher>/KEY /cmip5/nwsc/<operator>/KEY

Application’s keys(Dataset names publishing) (NLSR)

/cmip5/nwsc/<router>/KEY

Keys for ndn-atmos

Self-signed root key/cmip5/KEY

/cmip5/lbl/KEY /cmip5/nwsc/KEY… Site’s keys

/cmip5/lbl/<DataPublisher>/KEY /cmip5/nwsc/<operator>/KEY

Application’s keys

(Dataset names publishing) (NLSR)

/cmip5/nwsc/<router>/KEY

Trust Model

Only namespace owners are allowed to publish data

Data provenance built into the data packet

/PublisherA/publish

Publisher A’s signature

- /PublisherA/publish/file/1

+ /PublisherA/publish/file/3

Content Name

Signature

Data payload

Valid publish message

Trust Model

/PublisherA/publish

Content Name

Signature

Data payload

/PublisherA/publish

- /PublisherB/publish/file

Valid publish message Invalid publish message

Trust Model

/PublisherA/publish

Content Name

Signature

Data payload

/PublisherA/publish

Trust Model

/PublisherA/publish

Content Name

Signature

Data payload

/PublisherA/publish

Name Discovery

ConsumerCatalog

prefix>/query

Name Discovery

ConsumerCatalog

2) Query with parameters

(model=cmip5 AND frequency=6hr)

prefix>/query

Name Discovery

ConsumerCatalog

3) Query local DB; Packetize

results under

/<catalog-prefix>/query-

results/<params>

prefix>/query

Name Discovery

ConsumerCatalog

results under

results/<params>

3) ACK

prefix>/query

Name Discovery

ConsumerCatalog

results under

results/<params>

3) ACK

4) Fetch query results (name list)

prefix>/query

Name Discovery

ConsumerCatalog

results under

results/<params>

3) ACK

4) Fetch query results (name list)

prefix>/query

5) Fetch desired dataset(s) or

re-query

Data Publication

Catalog

Accept publish requests:

/<catalog-prefix>/publish

Authenticate and retrieve

data names from publisher

Sync names with other

catalogs

Publisher

Generate NDN names for

datasets/services

Inform catalog of names to

add/remove

PublisherCatalog

Data Publication

Catalog

catalogs

Publisher

datasets/services

add/remove

PublisherCatalogRequest publish

Data Publication

Catalog

catalogs

Publisher

datasets/services

add/remove

Fetch published name list

Data Publication

Catalog

catalogs

Publisher

datasets/services

add/remove

Fetch published name listValidate data name

against trust model

Data Publication

Catalog

catalogs

Publisher

datasets/services

add/remove

Fetch published name list

Share names with other

catalogs

Validate data name

against trust model

Name Discovery

Catalog

Accept queries on

/<catalog-prefix>/query

Query local DB

Packetize the returned names

results/<params>

Query catalog for names with

specified components

e.g.: model=cmip5 AND

frequency=6hr

Fetch generated name list

Fetch desired dataset(s) or re-

ConsumerCatalog

Name Discovery

Catalog

Accept queries on

Query local DB

results/<params>

frequency=6hr

ConsumerCatalogQuery with parameters

Name Discovery

Catalog

Accept queries on

Query local DB

results/<params>

frequency=6hr

Query local DB;

Packetize results

Name Discovery

Catalog

Accept queries on

Query local DB

results/<params>

frequency=6hr

Query local DB;

Packetize resultsACK

Name Discovery

Catalog

Accept queries on

Query local DB

results/<params>

frequency=6hr

Query local DB;

Fetch query results

Name Discovery

Catalog

Accept queries on

Query local DB

results/<params>

frequency=6hr

Query local DB;

Fetch data with

standard NDNFetch query results

Name Discovery Optimization

Catalog

Accept queries on

/<catalog-prefix>/queryParams

Query local DB

/<catalog-

prefix>/queryParams/seg#

In case of failure, queries get

redirected to another catalog

Consumers

Can query any catalog

instances

Can transparently failover to

another catalog

• Avoid maintaining state between user and catalog

• Enables graceful failover

Simplified xrootd Under NDN

NDN integrates discovery, failover, retrieval …

Provides a better abstraction to the applications

/my/file /my/file

Data Servers

Client

/my/file /my/file

Data Servers

Client

? /my/file

/my/file /my/file

Data Servers

Client

? /my/file

Name Discovery Challenges

Users may need to discover content/services without knowing a

the full NDN name prefix structure

NDN names are contiguous prefixes

Users may only know a few disjoint name components (e.g.

frequency=6hr)

But can not use wildcards for name discovery

Consumer

User wants: /CMIP5/output1/VA/6hr/2016

frequency=6hr)

Consumer

/CMIP5

frequency=6hr)

Consumer/CMIP5/output/BCC/6hr/1998

/CMIP5

frequency=6hr)

/CMIP5

/CMIP5/output/BCC/6hr (exclude 1998)

frequency=6hr)

/CMIP5

/CMIP5/output/BCC/6hr (exclude 1998)

May take too many requests to find desired data or service

NDN Support for Big Science

NDN Names separate data from hosts

Discovery: Names directly translate to network queries

Failover: Network can get verifiable data from anywhere

Retrieval: Data can be fetched from optimal source(s)

Investigate the use of NDN as a platform for scientific data

applications

Understand data management challenges of various scientific domains

Develop prototype applications to leverage NDN's built-in features

Use these applications as case studies to drive NDN research aspects

Summary

NDN improves scientific data management at scale

Apps benefit from transparent multipath, automatic failover, etc.

Built-in security provides publisher provenance

Names are the common building block for content and services

Names are flexible: can refer to static content or dynamic services

Catalog supports efficient publication, non-contiguous name

discovery

Users can discover content and services with minimal a priori knowledge

Catalog validates publication requests for authorization

Managing Scientific Data with NDN

Science testbed

10G testbed (courtesy of ESnet,

UCAR, and CSU Research LAN)

Nodes strategically located near

scientific data (climate +HEP)

CC-NIE NSF award

Distributed, synchronized catalog of

names and services

Common functionality: publishing,

discovery, access control, etc.

Search and retrieval UI

Platform for further research and

experimentation

Research questions:

Namespace construction, distributed

publishing, key management, UI design,

failover, etc.

Functional services such as subsetting

Mapping of name-based routing to

tunneling services (VPN, OSCARS, MPLS)

Managing Scientific Data with NDN

Science testbed

10G testbed (courtesy of ESnet,

UCAR, and CSU Research LAN)

CMIP5 and HEP data

CC-NIE NSF award

Name-based Internet architecture

Name the data, not the host

All data digitally signed

Unifies and pushes common functionality

to the network: publishing, discovery,

access control, etc.

Data Intensive applications

Automatic pervasive in-network caching,

parallel retrieval, automatic failover

and more

Simpler alternative middleware

implementation e.g., ESGF, xrootd

managing scientific data with ndn · 2015-09-29 · managing scientific data with ndn chengyu fan,...

Documents

ndn-mag-april 2014

ndn software description

networking ndn oct 2010 2

ndn membrane software

security measures of ndn

ndn managed gateways and the ndn testbed

open iot/ndn research challenges

memory designing using josephson gates susmit biswas...

april 08 ndn p 1-12

for personal use only - asx · susmit shah kevin thomson...

ndn-android: ndn networking stack for android...

ndn-ss- april 2014

by: roman olschanowsky roman2u@sdsc.edu an introduction to...

sept ndn 13-23 final

thoracic trauma ndn (1)

ndn pd day yarnall 2013

strategic environmental assessment (sea) and a...

design of ipv4/ipv6 transition strategy based on ndn ·...

routeviews + bgpmon enabling bgp monitoring and analysis...

2010 ndn firemen special pages