Improved Access to RDA from the MSS

OSD Executive MeetingApril 28, 2009

Current situation

RDA, modes of accessMSS to NCAR connected machinesSAN to the networked world (35 TB only)Personalized requests to DSS staff, to the world

Ad hoc, one-off efforts, no automation, less than optimal metrics, requires strong initiative for users

Quick drive through Illustrate foundations for the new service

Registration

To get RDA data a user MUST registerMetadata and searching are freely openDB controlledLevels of registration control access

Non-restricted data (first level)Restricted access requires additional registration, e.g.

agreement to special usage condition for JRA-25

Access offerings

MSS File Selection/Information

Example: ECMWF, May 2008

Script to Read MSS files

Enable this process for users that don’t have NCAR MSS access – outside users

RDA Dataset Request System (DSRQST)

Written project description and requirements Highlight of requirements in three categories

UsersDSS ControlSystems

User Requirements

Prioritized Service Levels UCAR, U.S. Univ, U.S Gov. etc

Only qualified users see DSRQST interface All data restrictions strictly enforced Download options matched RDA online

Point and click ‘wget’ script

Automatic notification when data is ready Password protected download directories Unreasonable requests – pop-up window including fault

description and options Most likely too much data

DSS Control Requirements

System workflow settings1. Staff inspection of request, approval and start2. Fully automatic – staff still receives all workflow information

Implementable at the dataset level Controlled initiation

Service initiated by responsible staff Maximum download amount controlled at dataset level

(~ default 20-100 GB) ‘msrcp’ MSS files out to network attached systems Set priority process levels, 1-10

System Requirements

Daemon controls number of processes Throttle impact on MSS Monitor temporary storage area

Automatic scrubbing of staged data Default 5 days, more by request

File transfers stops if temporary storage is near full All metrics logged as “one-off” requests Cross-check file availability from SAN

Don’t transfer from MSS is already available online Corollary: share files between requests

Restart capability – mid-stream recovery End-to-end control from DB settings

Next steps

Coding is underway, datasets for prototype are identifiedWhen – 1-2 months

Implement data processing on files – data product preparationPossible – split request and use multiple processes Utilize CISL high-end computing

Implement product write service onto the MSS for NCAR usersShort retention period

End

Questions

Complex systemDB – MySQLMSS read and write interactionshtmlphp java scriptperl