web portal design workshop, boulder (co), jan 2003 luca cinquini (ncar, esg) the esg and ncar web...

Luca Cinquini (NCAR, ESG)

Web Portal Design Workshop, Boulder (CO), Jan 2003

The ESG and NCAR Web Portals

Luca CinquiniNCAR, ESG

Outline:1. ESG Data Services Strategy2. ESG Prototype Web Portal demo3. NCAR Community Data Portal4. NCAR Prototype Search & Discovery engine5. Final comments



ORNL: Climate storage &computational resources

ORNL: Climate storage &computational resources

LANL: Next generationcoupled models & computing

LANL: Next generationcoupled models & computing

ANL: Computational grids,& grid-based applications

ANL: Computational grids,& grid-based applications

USC/ISI: Computational grids,& grid-based applications

USC/ISI: Computational grids,& grid-based applications

NCAR: Climate changepredication and scenarios

NCAR: Climate changepredication and scenarios

LBNL: Climate storage facility

LBNL: Climate storage facility

LLNL: Model diagnostics& inter-comparison

LLNL: Model diagnostics& inter-comparison

The Earth System Grid




• “Synergistic collaboration” among several US national labs and research centers (ANL, ISI, LBNL, LLNL, NCAR, ORNL) involved in atmospheric science and scientific computing

• 3 year project funded by the DOE Scientific Discovery through Advanced Computing (SciDAC)

• Goal: build the next generation computational and data management environment for the geosciences: “a system of geographically distributed data and computational centers which will allow seamless access for earth scientists to data repositories, analysis tools and computational resources”

• Strategy: application and extension of Grid technologies (and other state of the art IT) to the geosciences

• Initial focus on next generation climate modeled data (PCM/CCSM)



ESG Goal Functionality

• Security (authentication/autherization mechanisms, management of security credentials)

• High speed, reliable file transfer between data centers (both to/from local and remote storage)

• Management and use of scientific metadata (generation, storage, update, annotation, query)

• Data search and discovery through metadata• Virtual data management (aggregation/subsetting of

physical data files, transfer and access of data containers)• Data analysis and visualization• Resources monitoring and discovery• Activity/usage logging and monitoring



ESG Technology Strategy

• Use a core library of state of the art, powerful technologies (Globus, Java, XML, servlets, web services, Python)

• Accomplish desired functionality within modular, extensible software components (or services)

• Integrate components in powerful, easy to use, end to end applications

• Provide both descrete (web/grid services) and centralized (web portal) access to the applications



Core software library(Globus, Java packages, CDAT, NCL, etc.)

Core software library(Globus, Java packages, CDAT, NCL, etc.)

ESG Software Architecture

TransportTransport SecuritySecurity MetadataMetadata

AnalysisAnalysis VisualizationVisualization MonitoringMonitoring

ESG Data

Services

Web PortalServlets/JSP

Web PortalServlets/JSP Web/Grid ServicesWeb/Grid Services RMI serversRMI servers

Web clientWeb client Web service clientWeb service client RMI clientRMI client



ESG Super Computing 2002 Demo

Prototype of a unified web portal that integrates powerful, different technologies in a unified flow that constitutes and

example of a scientific activity session

Data Search & Discovery(RLS, MCS)

Data Search & Discovery(RLS, MCS)

Authentication(GSI)

Authentication(GSI)

Data SelectionData Selection

Data Transfer(HRM)

Data Transfer(HRM)

Data Visualization(LAS)

Data Visualization(LAS)

Data SubsettingData Subsetting

Authorization (CAS)Authorization (CAS)

Data Transfer (openDAPg)Data Transfer (openDAPg)



ESG SC 2002 DEMO



TOMCATServlet engine

TOMCATServlet engine

MCSMetadata Cataloguing Services

MCSMetadata Cataloguing Services

RLSReplica Location Services

RLSReplica Location Services

SOAP

RMI

MyProxyserver

MyProxyserver

MCS client

RLS client

MyProxy client GRAMgatekeeper

GRAMgatekeeper

CASCommunity Authorization Services

CASCommunity Authorization Services

CAS client

diskMSS

Mass Storage System

HPSSHigh PerformanceStorage System

disk

HPSSHigh PerformanceStorage System

disk

disk

SRMStorage Resource

Management

SRMStorage Resource

Management

SRMStorage Resource

Management

SRMStorage Resource

Management

SRMStorage Resource

Management

SRMStorage Resource

Management

SRMStorage Resource

Management

SRMStorage Resource

Management

gridFTP

gridFTP

gridFTPserver

gridFTPserver

gridFTPserver

gridFTPserver gridFTP

server

gridFTPserver

gridFTPserver

gridFTPserver

openDAPgserver

openDAPgserver

CAS-enabledStriped-gridFTP

server


server

LBNL

LLNL

ISI

NCAR

ORNL

ANL


Striped gridFTPclient

Striped gridFTPclient

gridFTP

openDAPgserver

openDAPgserver


server


server

gridFTP

openDAPgserver

openDAPgserver


server


server

gridFTP

LASLive

AccessServer

LASLive

AccessServer



The NCAR Community Data Portal

• Internal NCAR project (SCD, ATD, CGD, Unidata, JOSS, HAO, DLESE) aimed at providing a central web portal for all of UCAR/NCAR data assets

• Data scope much larger than ESG: topic (model data, experimental, measurement, satellite), format (netCDF, HDF, Grib, etc.), amount (Pb), requirements (for analysis, access, metadata, etc.), user community

• But goal is at least initially narrower: initial work has focused on: Data Visualization

LAS, DODS servers connected to Ferret, NCL or CDAT engines allow visualization of selected datasets

Metadata Experimenting with THREDDS schema for data cataloguing Developing Scientific Metadata Services: a system of

software components that would allow management and use of metadata (generation, storage, update, query, browsing)

Initial prototype: “Google”-like search and discovery of data

THREDDS CATALOG

<dataset ID=“…”> <metadata href=“..” type=“..”> <dataset ID=“…”> <metadata href=“…” type=“…”> </dataset></dataset>

THREDDS CATALOG

<dataset ID=“…”> <metadata href=“..” type=“..”> <dataset ID=“…”> <metadata href=“…” type=“…”> </dataset></dataset>

Dataset metadata (DC)Dataset metadata (DC)

Dataset metadata (NcML)Dataset metadata (NcML)

Dataset metadata (other)Dataset metadata (other)

XML Native Database(Apache Xindice)

XML Native Database(Apache Xindice)

Collection: /thredds (name=ID)

Collection: /dc (name=ID)

Collection: /nc (name=ID)

Collection: /default (name=ID)

HARVESTING WEB SERVICEHARVESTING WEB SERVICE

Relational Database(Mysql)

Relational Database(Mysql)

Table: DATASET (ID, key)

Table: TEXT ( key, name, value, ns, qualifiers)

Table: FLOAT ( key, name, value, ns, qualifiers)

Table: DATETIME ( key, name, value, ns, qualifiers)

SCIENTIFIC METADATA SERVICESQuery Architecture

EXTRACTOR APPLICATIONEXTRACTOR APPLICATION METADATA QUERY SERVICE

METADATA QUERY SERVICE

Generic Search & Discovery Query

(SQL)

Generic Search & Discovery Query

(SQL)

Text query

Numeric query (ex: geograpchic)

Temporal query

Specialized Query(Xpath, Xquery)

Specialized Query(Xpath, Xquery)

List ofDataset

IDs

List ofDataset

IDs

XMLdocument

XMLdocument

DC XSLDC XSL

NC XSLNC XSL

THREDDS XSLTHREDDS XSL

GENERIC XSLGENERIC XSL



CDP SEARCH AND DISCOVERY DEMO



A model for a collaborative architecture

• Use MVC software design model for web development: Model: core software libraries/packages where the business

logic takes place (e.g., HRM, CDAT, NCL, openDAP). View: user interface, how data is presented to the user Controller: mediates interaction between Model and View,

i.e. parses input request from View, invokes the corresponding methods on the Model, returns output to the View

• Additionally, integrate MVC applications by sharing data beans associated with the user session



RLS serverRLS server

RLS APIRLS API

RLS web clientRLS web client

RLS web UIRLS web UI

HRM serverHRM server

HRM APIHRM API

HRM web clientHRM web client

HRM web UIHRM web UI

datadata

(Java beans)

MODEL(any language,

Java API)

CONTROLLER(Java servlets,

JSP taglibs,Java classes)

VIEW(JSP)

Integrated MVC architecture example

web portal design workshop, boulder (co), jan 2003 luca cinquini (ncar, esg) the esg and ncar web...

Documents