1 class – simple noaa archive access portal snaap eric kihn and rob prentice noaa/ngdc esip...

1CLASS –Simple NOAA Archive Access Portal

SNAAP Eric Kihn and Rob Prentice

NOAA/NGDCESIP Meeting

January 7th, 2009

Simple NOAA Archive Access Portal


SNAAP Goals

Integrate diverse data archive systems via a standards based, user focused, web services interface

Create a set of tools that shows the benefit of such an API for users

Discover new technology and evaluate cutting edge tools for handling integration and presentation

Document integration of multiple data types and sources through an single API (time-series, grid, swath, etc..)

Keep it simple. Anything tough is a client problem


The Data Environment Now

CLASS ADS(e.g. AVHRR)

Web Interface Subscriptions

Human Machine

DataProviders

Visualization Inventory

Order

NM

MR

/Metadata

NGDC/SPIDR(e.g. DMSP, Space Physics)

Web Interface

Web Interface

Metadata

SPIDR WS

SNAAP

Web Interface

Metadata

Data Access


What is the origin of SNAAP?

The project evolved from the goal to put an API on CLASS. It was quickly realized that the boundary is then arbitrary and it makes sense to extend the API to other data systems

CLASS (and other systems) already have a functional API for intra-system communication but the duplicity of systems causes real headaches

The system is currently limited to archive (non-real time) data systems following its archive access origins

SNAAP is part of the GEO/GEOSS effort. Its architecture and capabilities do for NOAA what is being done in GEOSS (archive)


SNAAP Functionality Search for appropriate data sets

Discover data by time, type, location, abstract, etc. Support for multiple metadata schema (FGDC, IEEE, SPASE) Standards based (xQuery and REST) This is in already in CLASS (4.3.3) – awaiting policy

Order Data Needs a standard way to describe ordering options This is done via Ordering Extensions (OE) which are linked to and extend

standard metadata Needs some framework for service integration (i.e. transforms) this is

provided via (Data Access and Integration) DAI Actions

Provider Integration Layer We have chosen OGSA-DAI for the demo (This could change in production) We offer both a SOAP and REST interface (slightly different functionality) Its important that the backend retrieval not effect the consumer.


Advantages

This approach mirrors that of the highly successful OGC specification for map services (what, make, retrieve)

It minimizes entry burden for new providers

Its simplicity makes it scalable

It doesn’t preclude or interfere with on-going activity, just opens up new integration possibilities.


Metadata: A Complete Data Environment is More than Just the Bits

FGDC records

Ordering Extentions

Wiki Documents

Presentations

SEARCH in CLASS

metadata

Search result:

ResourceID_1ResourceID_2ResourceID_3

FGDC Metadata

OE (1…*)

User Guide

Slideshow

CLASS ResourceID

SNAAP metadata Virtual Observatory

SNAAP Web Application

SNAAP API Data Sources

CLASS

CLASS products

Visualization service

Inventory service

Order service

OGSA-DAI for CLASS (Globus Toolkit 4)

Plugin

Data Request REST API for CLASS

OGSA-DAI client toolkit


Snapshot of the SNAAP API architecture

CLASS

CLASS products

Visualization service

SPIDRESG - INE - IDEAS – ESSE

NCEP , ERA-40 , ...

DB cluster

Inventory service

Order service

WebServices

OPeNDAP

OGSA-DAI for CLASS (Globus Toolkit 4)

Plugin PluginPluginPlugin

NOMADS

NARR

NetCDFNetCDF

NetCDF

THREDDS(OPeNDAP)

WCS(OGC)

SABR

OLS DMPS

Previewimage

Plugin

Inventory

GranuleGranuleSatellite

Granule

CLASS web interfaceCLASS

metadata

REST API for CLASS

DB cluster

OGSA-DAI client toolkitGRID client toolkit


Why OGSA-DAI?

Standard tool in the Grid community

Includes a “process” for extraction

Built in support for asynchronous transactions

Includes support for domain login

Looked at alternatives like OpenDap, WCS, Custom applications –documentation of our analysis is available

Problem: Its very complex and not well accepted in NOAA


Data Order Flow

OE Web Form Servlet

XSLT get

ClientOGSA-

DAI

XML Result

Get DataProcess

Mine

SQLXML

GranuleTime

Series

12

Adapter

Storage

ErrorMessage

Data

Ser

ver

Process Document via SOAP

3


CLASS Ordering Extensions (OE) schema upper level


CLASS OE schema: parameters and time


CLASS OE schema: map and ROI


CLASS OE schema: station map


Key Goal:Minimize the Burden on Providers

Create a standard metadata record for the data set. (They should be doing this anyway).

Create an Ordering extension document. If they are offering web distribution this is trivial.

Write an adapter to the middle layer, unless they have a supported service (like JDBC, FTP, etc..) in which case adapt an existing one.

To hook to SNAAP a provider would :

The provider then:

Continues to serve data and metadata as before plus SNAAP access.


Data Sets and Types

Data Types Time-series – Sunspot number Grids – NCEP Reanalysis Stations – Ionospheric Soundings Swath - AVHRR Profiles – Ocean Profile Maps – Nighttime lights

More?


Target Audience

There is a clear understanding that many communities are already well served by existing data systems

The SNAAP API targets developers who want to add new functionality to integrated NOAA archives

When operational it will provide fully redundant support for data discovery and retrieval

It is hoped that by having the data accessible in a standard open way more innovative client tools will become available.


Visualization screens – 1time series and map server


Visualization screens – 2NASA World Wind plugins


MS Excel and NASA World Windinteroperability for trajectory data

Excel with track points

Excel with data along tracksDrag & Drop controls

NASA World Wind


Future Plans

Plan transition to operations (CLASS Team) This will identify a CLASS build and date for SNAAP inclusion This would require some review process (including outside)

and a versioning of OE Seek outside review of the OE documentation.

Plan to include non-granule based asynchronus (i.e. models, transforms) data extractions

Enhance the connection with GEO-IDE

Upgrade to DAI 3.0 - DONE

Expand proto-type to cover more NOAA and non-NOAA data types and services and document lessons learned

Release the work to the public: Metadata tool is available as VxOware http://sourceforge.net/projects/vxoware/

Work on single user sign-on.


BACK UP SLIDES


Asynchronous Ordering

CLASS API Demonstration


Orbit Inventory List


User Basket (Order & Check-Status)


Order System Entities

•Granule List now using ECHO XML in the return•New entity “Data Basket” – handle to collection of order•Demo currently orders all data centrally will need to be adapted


Order Sequence Diagram

•Data basket ID is the API handle•OGSA-DAI activities handle the extraction and packaging•Status check is currently by basket only (XML return)•Finished granules return a URL handle to the entity