status of the ghrsst gdac (global data assembly center) at the jpl po.daac ed armstrong, jorge...
Post on 26-Mar-2015
213 Views
Preview:
TRANSCRIPT
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Status of the GHRSST GDAC (Global Data Assembly Center) at the JPL PO.DAAC
Ed Armstrong, Jorge Vazquez and Andrew Binghamhttp://ghrsst.jpl.nasa.gov
28 Nov 2005 US GHRSST meeting, RSMAS
Overview of GDAC Tasks
Ingest project data filesL2P, L4, HR-DDS, MMR and MDB files from RDACs
Stage on public FTP site for 40 daysDeliver data and metadata to LTSRF (NODC) for 30 day old dataMaintain metadata in a relational database (MMR)Provide web-base interface for data discovery and metadata queryFill ancillary fields for all L2P productsSupport GHRSST User Services and ApplicationsMaintain a match-up database of in situ/satellite obs RDAC: MODIS L2P production
. . . .data management and distribution center for all GHRSST products
28 Nov 2005 US GHRSST meeting, RSMAS
Recent Activities
Public GHRSST data release in August 2005Interacted with OBPG and RSMAS to design the MODIS L2P processing architecture New Linux-based web and database severs, and development workstation integrated into PO.DAACGDAC web pages enhancedAccess to MMR improvedInput for GDS rev 1.7 including metadata revisionIDL and C readers for L2P Implemented FGDC metadata and transfer to LTSRFIngested NWP and model data streams for L2P ancillary fieldsDeveloping code and methodology for realtime ancillary filling Implemented user support through a customer management tool (RightNow)First draft of a user requirements document
28 Nov 2005 US GHRSST meeting, RSMAS
GDAC System Architecture
GDAC (JPL U.S.)
RDACs
OCEANIDS(Data Management)
MMR/MDB
L4PC(MIST – U.S.)
Archive(NODC – U.S.)
NRTUsers
Users
Medspiration (Europe)
RSS(U.S.)
BlueLink(Australia)
PO.DAAC(U.S.)
NGSST(Japan)
NAVOCEANO(U.S.)
Data Tools
28 Nov 2005 US GHRSST meeting, RSMAS
GDAC Web Page
28 Nov 2005 US GHRSST meeting, RSMAS
Master Metadata Repository
28 Nov 2005 US GHRSST meeting, RSMAS
Data Access
FTP: podaac.jpl.nasa.gov/pub/GHRSSTData public since late August 2005Organised by:
datatype / sensor / RDAC / year / DOY
Search interface through the MMR
Future delivery mechanismsPOET (L2 subsetting)OpenDAPLive Access ServerSubscriptionData casting
28 Nov 2005 US GHRSST meeting, RSMAS
Data IngestRDAC Datastream StatusEUR L2P TMI
L2P SEVRIL2P NAR 16 & 17 & 18L2P AVHRR LAC (16 & 17)L2P AVHRR GAC (16 & 17)L2P ATSRL2P AMSR-EL4 UHFndHR-DDS
GoodGoodGoodGoodGoodGoodGoodGoodNo data since 25 Mar
REMSS L2P AMSR-EL2P TMI
Good, sporadic in Oct
NAVO L2P AVHRR LAC (18)L2P AVHRR GAC (18)
Expected Dec 2005 ?
RSMAS/OBPG/JPL
L2P MODIS_A In preparation
NOAA NESDIS L2P GOES Unknown
AUS (Bluelink)
L2P AVHRR HRPT Unknown
28 Nov 2005 US GHRSST meeting, RSMAS
Data Latencies
Datastream Avg (hr)
Std Dev (hr)
Max (hr)
EUR L2P TMIEUR L2P SEVRIEUR L2P NAR 16EUR L2P NAR 17EUR L2P AVHRR LAC 16EUR L2P AVHRR LAC 17 EUR L2P AVHRR GAC 16EUR L2P AVHRR GAC 17EUR L2P AATSREUR L2P AMSR-EEUR L4 UHFnd
20.77.27.47.08.68.57.58.19.422.211.8
40.416.815.112.013.312.211.213.814.648.113.4
137639216998108971033883731377108
28 Nov 2005 US GHRSST meeting, RSMAS
Oceanids ingest statistics
1 2 3 4 5 6 7 8 9 10 11
S1
02000
4000
6000
8000
10000
12000
14000
16000
18000
month - 2005
granules
L2P/L4/HRDDS granules ingested77000 total
Granules
Volume (GB)
1 2 3 4 5 6 7 8 9 10 11
S1
0
1
2
3
4
5
6
7
month -2005
volume - GB
Data volume ingested47 GB total
28 Nov 2005 US GHRSST meeting, RSMAS
Ancillary fields JPL GDAC will fill the L2P ancillary fields of wind, solar insolation, sea ice and aerosols
Some RDACs cannot fill these fields in an operational environmentGDAC has accepted this additional responsibility at last GHRSST ST meeting
Ingest numerical weather prediction and model data, i.e., NCEP, NAAPS, FNMOC sea ice and SST
A backup data stream will be implemented for each ancillary field (e.g., ECMWF model output)Ancillary filling methodology and management has been conceptualized.
• Readers for input fields being built
28 Nov 2005 US GHRSST meeting, RSMAS
NOAA LTSRF Interface Status
After 30 days data are placed on a rolling stage for NOAA to pull FGDC metadata created “on the fly” through access to the MMR
Formatted for the NOAA accession record (daily collection of same product files)
Currently NOAA has implemented an automated retrieval of L2P data and FGDC metadataFGDC records for all L2P Medspiration data created through Nov 2005
Other products to follow
28 Nov 2005 US GHRSST meeting, RSMAS
MMR Status
MMR (Master Metadata Repository)Fully transactional MySQL relational databaseSchema finalized and implemented for FR and DSD metadata in Jan 2005
• New database server implemented in Jun 2005
Over 80,000 DSD and FR records ingested since Feb 2005
• No operational failures
Web search interface improved, access is very fast.Automated translation to FGDC metadata through XML style sheet conversion
• Other standards can also be supported
28 Nov 2005 US GHRSST meeting, RSMAS
User services and applications
Updated US GDAC User Requirements document
GHRSST requirementsGDAC system requirementsRDAC requirementsUser requirements
• E.g., OOSes
28 Nov 2005 US GHRSST meeting, RSMAS
MODIS L2P
Joint effort between Univ. of Miami RSMAS, NASA Ocean Biology Processing Group (OBPG) and JPL
This is a new RDAC activityInitially concentrate on Aqua SST
RDAC Division of LaborRSMAS: Algorithm and error statisticsOBPG: 1 km L2 SST production with some ancillary fieldsJPL GDAC: L2P integration with error statistics and ancillary fields
Current StatusOBPG has produced a realtime L2 SST stream for MODIS Aqua (and Terra) RSMAS has developed a MODIS error “hypercube”JPL has tested throughput and acquisition from OPBG server
28 Nov 2005 US GHRSST meeting, RSMAS
Future activities . . .
Ancillary fields fillingAccept delivery of Linux 8 node clusterInstalled in Jan 2006Implement code for realtime filling
• Investigating parallelization of processing
MODIS L2PFinalize transfer between three partiesImplement error statistics and L2PSchedule for Aqua and Terra SST L2P
Interface to new RDACsNAVONOAA NESDISOBPGUpdate ICD
28 Nov 2005 US GHRSST meeting, RSMAS
. . . Future activities
FGDC metadataOperational transfers for all GHRSST products (L2P, L4) to LTSRFs
User ServicesGHRSST product supportApplications development
Data deliveryL2P subsetting in POETOpenDAPImproved search and delivery via MMR
New technology infusionGDAC will be a test bed for the Earth Science Data Casting Project
• a recently funded NASA ROSES activity for XML-based data management and delivery
Implement the Matchup Database (MDB) with similar functionality as MMR
top related