Ice Bridge Data Management

Scope, Challenges, SolutionsRon Weaver, NSIDC DAAC Manager

National Snow and Ice Data CenterSupporting Cryospheric Research Since 1976

IceBridge Data Management. September 20092

Topics

• Vision, Guiding Principles• Science Goals• Data Management Goals

In Response to Science Goals• Proposed Activities• Schedule• Challenges• Questions• Next Steps

IceBridge Data Management. September 20094

Science Goals - IceBridge

• Ice Sheet Surface elevation data after ICESat-1 mission ends Areas of rapid ice sheet change, Sea ice mass estimates

• Computed measurements critical to ice models Bed topography, grounding lines, ice and snow thickness

• Link ICESat-1 to ICESat-2 through baseline inter-calibration measurements

• Campaigns to support sea ice (thickness, type, extent)• Technology incubator and test-bed

Airborne and satellite sensors, UAV, photon counting, cross-track measurements

IceBridge Data Management. September 20095

Data System (at NSIDC) Goals

1. Data system supports science goals and user needs

2. Provide appropriate and adaptable data services and tools for these data

3. Provide (and/or promote) understandable, accessible, and relevant data products to broader science community

4. Manage data sets from IceBridge, ICESat-1 and WFF ATM (via funded ACCESS project) in a coherent and interoperable manner

5. Prototype the ICESat-2 data system6. Data Stewardship

Data Types

L0

L1

L2

L3

LidarsATM GLAS LVIS UAF

LiDARIce Radars

PARIS KU-IceKu-Snow ……Gravity Magnetics

Profiles/SwathPoints - Waveforms

Point cloud

Waveform

DEM DEMsurface

Profiles/SwathReturn Energy -

Waveforms

Ice in-situlayers

SnowAccum.

DEMbed

profiles profiles

GravityPotentialsurface

AeromagPotentialsurface

Surface mapsSea ice – ice sheets

Ice sheet roughness

External Products and ServicesAg

greg

ation

Ser

vice

sModels

Dat

a Pr

oces

sing

Lev

els

Services Data CastingsTools

USERS

ICECAP

ICECAP LDEO

Task Sequencing – After Preliminary Discussions With NASA HQ

ESTABLISH OIB PROJECT TEAM•Coordinate DAAC OIB data mgmt. activities at NSIDC•Coordinate NSIDC activities with:

• Science team• Instrument teams• ESDIS and HQ

CONDUCT DATA ASSESSMENT•Establish OIB initial baseline data list•Characterize existing data formats, metadata processes, services, etc.•Determine initial LOS for all OIB data sets•Assess tools and services needed by OIB Science and Instrument Teams and User Community•Provide NASA HQ with work plan with cost estimates for remaining work

BROKER OIB DATA SETS•Metadata entries in NSIDC data catalog with LOS designated•Collect/extract/reformat documentation and metadata•Metadata entries transferred to ECHO and GCMD

ESTABLISH LIST OF DATA FOR INGEST AT NSIDC

•Determine LOS and resources needed for transfer•Negotiate agreements with Instrument Team(s)•Determine transfer phasing

EXTEND TOOLS AND SERVICES TO DATA AND PRODUCTS

•Determine phasing of development efforts

IMPLEMENT OIB DATA PORTAL•Rapidly implement focused entry point for OIB data and services•From metadata entries in NSIDC and ECHO data catalogs with LOS designated•Reuse/Augment SearchLight, CCAWS/SAGE technologies

IMPROVE INSTRUMENT TEAM DATA MANAGEMENT PRACTICES

•Consult with Instrument Teams

IMPLEMENT DATA AGGREGATION SERVICES

INGEST DATA INTO NSIDC VO OR ECS SYSTEMS

•Phased ingest based on established priority

A C

D

E

F

H

I

G

B

PROVIDE APPROPRIATE DATA PRODUCTS DERIVED FROM

INGEST DATA STREAMS

Current NSIDC Schematic Architecture

Ingest Paths Archival SystemsData & Metadata

Production Interfaces & GUIs

Services

EOS Satellite Data

V0 Ingest Paths

PDR Server

PMIngest

SearchlightIngest (TBD)

PMProduction

Manual Ingest

WIST

DPInterface

Machine to machine

Sub setters

DataPool SAN

ECS

ECHO

V0 SAN

V0 MDDB (to be replaced)

SearchLight

OIB Portal

Legacy GUIs

SAGE CCAWS

GUI

Service Casting

SLight GUI

Lines indicate data and metadata flowsDotted outlines are functions under developmentGrayed out items are to be replacedRose colored items are proposed

IceBridge Data Management. September 200914

Schedule• HQ decision on NSIDC Proposal (November?)

scope, level of effort, schedule• NSIDC appoint a project leader and identify team

members (January?) (Proj Mgr, scientist, USO, OPS, Engr) probably aggregate 1-1.5 fte level of effort

• Implement approved tasks Work on integrated data products ?Conduct Data Assessment (prelim report March

2010?)

• Iterate with NASA over requirements

IceBridge Data Management. September 200915

Challenges• Timeliness Of Data Delivery• Understanding And Responding To User Community Needs• Complexity Of Data Streams, Especially Experimental

Instrumentation Changing Formats Of Experimental Data Dataset Co-registration

• Appropriate Data Set And System Integration Develop Tools And Services To Fit Complex Data Or Adapt Complex Data To Known Tools And Services

• Distributed (Virtual?) Data Management Systems Who Holds What Data? Who Decides? How Does Interoperability

Get Accomplished?

Note That Archive Volume Is Not On This List

BACKUP

IceBridge Data Management. September 200918

It will use NSIDC’s SearchlightUSERS

JAZ

Struts 2 Action Objects(controller)

WCS Service

ExternalDatabase

MetadataDatabase

JPA Hibernate

ECSFile System

ECHOMetadata

NSIDC FileSystem

Service Objects

Repository Objects

Domain Objects

BrokeredDatabase

Workflow Objects

WMS Service WFS Service

Analysis Objects

CCAWS UIG-CAP OGC UI

External Metadata

SearchLight JSP Pages(view)

OPeNDAP ServerOther Systems

AJAX Libraries (DOJO, YUI)

SearchLight

CCAWS

G-CAPUser Interface Layer

Service Layer

Domain (model) Layer

Infrastructure Layer

Other Systems

Core Software Architecture

IceBridge Data Management. September 200919

Searchlight Components