ceos/wgiss 20, kyev, september 12-16, 2005 1 wtf-ceop implementation plan #1 status (wtf-ceop first...

CEOS/WGISS 20, Kyev, September 12-16, 2005 1

WTF-CEOPImplementation Plan #1 Status(WTF-CEOP first prototype, by JAXA)

September 12, 2005

Osamu Ochiai - JAXA

Ben Burford - JAXA/RESTEC


• Implementation Plan #1 Review

• Implementation Plan #1 Status

• Status of archives (In-situ, MO, Satellite)

• Adding additional functions

• CEOP GCMD Portal

• Documentation and manuals

• Conclusions and JAXA Future Plan

Table of Contents


WTF-CEOP IP #1

• Provide access to CEOP project data (data integration)– In-situ (archive at UCAR)– MOLTS (archive at MPI)– Satellite (archive at Univ. of Tokyo)– Model Output (archive at MPI)

• Basic functions– Access data from archive sites– View data (plots, view data values on screen)– Download (e.g. NetCDF format)


WTF-CEOP IP #1

• “Use existing software”– Live Access Server (LAS)– DODS/OPeNDAP server

• Advantages/capabilities– Detailed access (“6 way subsetting”)– Flexible – can be modified in several ways– Analysis capabilities/services can be added– Well documented– Good support – users group and developers


WTF-CEOP by JAXA

Live Access Server(LAS)

Ferret

Menus – Select DataFunctions – Plot, Display Download, etc.

　　　　　　UsersWeb Browser CSDIC Archive

Satellite

(Japan)

DODS

UCAR Archive

In-situ

(U.S.)

DODS

MPI Archive

MOLTS, Model Output

(Germany)

DODS

Coordination:

1. Selection, modification and test of OPeNDAP Freeform server

2. Coordination of DODS server installation at UCAR, MPI (not yet) and CSDIC (tuning)

IP #1/JAXA PrototypeSystem Overview

Development:

1. Develop “User Interface” menus (XML files, converted to populate MySql database)

2. Modify “Data Server” software (to access various CEOP data types on OPeNDAP Freeform server)









Table of Contents


Implementation Plan #1

Status

Last Fiscal Year Activities (ended March 31, 2005)

• Select data type• Select location (Reference Site, Station name)• Select variable name• Select by height or depth• Select by time range (start time / stop time)• “See the data” (plot and view values on the screen) • Download data (NetCDF, ascii or spreadsheet format) • Registration (user log on and access logs, to satisfy CEOP data policy)


Implementation Plan #1

Current Work

This Fiscal Year Activities (final - ends March 31, 2006)• Upgrade LAS software (from version 6.3 to 6.5)• In-situ data (update menus, add download of QC flags)• MOLTS (access by HTTP string from new MPI interface)• Satellite (band selection, period selection, download multiple scenes)• Model Output (access by HTTP string, convert from GRIB to NetCDF)• Choose to be notified by email when data is ready• Automate update of menus as new data is added• Expand User’s Manual• Write Maintenance Manual• Update and increase DIFs in WTF-CEOP portal in GCMD

Additional R&D, beginning this Fiscal Year• Presented by Osamu


Upgrade LAS Software:From Version 6.3 to Version 6.5

• Contains a prototype THREDDS Catalog • New Browser Management

– A new XML file (las/server/browsers.xml) contains a list of known browsers with an indication of whether they are compatible with the applet or not.

• New <ui_timeout> Property– The <ui_timeout> property is the number of seconds the user interface serv

er should wait for a product request to finish before 'backgrounding' the request.

– If the requested product finishes in less than <ui_timeout> seconds it is simply returned when ready. Otherwise, an HTML page is returned that allows the end user to 1) check on the status of this request; 2) ask to be notified by

email when the request is finished; or 3) cancel the request.









Table of Contents


In-situ Data Archive Development Status

• Archive at UCAR (Boulder, Colorado, US)• OPeNDAP Freeform Server was implemented• In-situ data – Surface, Tower, Soil, Flux (time s

eries)• All current in-situ data is online• Continuing – update menus as new in-situ data

is added at UCAR• New capability – include QC flags with data in d

ata download


In-situ Data - 7 QC Flags

G Good

B Bad

U Unchecked

I Interpolated/Estimated

C Reported value exceeds output format field size OR was negative precipitation.

D Questionable/Dubious

M Parameter value missing OR Derived parameter can not be computed.


MOLTS Data Archive(Development Status)

• MPI (Max Planck Institute), M&D (Hamburg, Germany) – MOLTS archive center for CEOP

• MPI decided to ingest MOLTS data into Oracle database– Develop Java routines or install DODS server– Request using HTTP string

• CEOP telecon on MOLTS data format– MPI will convert all (10 agencies) MOLTS data to NetCDF forma

t.

• 3 copies of MOLTS data (original, netCDF, Oracle).• MPI upgraded Oracle software during August.• MPI beginning development of MPI/WTF-CEOP interface

for MOLTS data in September (this month).


Model Output Data Archive(Development Status)

• MPI (Max Planck Institute), M&D (Hamburg, Germany) – Model Output archive center for CEOP

• MPI will reprocess all model output data (10 agencies, about 50 Terrabytes) to individual variables at individual levels– Some sample files are available

• WTF-CEOP access will be via HTTP string using MPI “Jblob” interface.

• WTF-CEOP will receive model output data in GRIB format, and must convert from GRIB format to netCDF format.

• MPI beginning development of MPI/WTF-CEOP interface for model output data in September (this month).


Satellite Data Archive (Development Status)

• CSDIC (University of Tokyo, Japan)– Satellite data archive for CEOP

• Working on installing Freeform Server at CSDIC• Upgraded libraries on CSDIC computer• Changed environment variables for HTTP string access via

Apache server to DODS server• Problem – now encountering DODS internal error• Satellite data format has changed (separate header and data

files)• Band selection is being added – select one band or all bands.• Time period selection is being added – all scenes in a time

period.• Package download function is being added – download is for all

scenes in a time period.









Table of Contents


Additional Functions

• Add background processing capability– “Wait for processing to finish, or be notified by email when file

is ready to be picked up from server?”• Write utilities to automate updating of WTF-CEOP

menus as more data is added– Possibly use metadata provided by UCAR to update WTF-

CEOP menus– Especially satellite data (over 1,000,000 scenes of satellite

data)– Read satellite data directories and create XML file with menu

contents









Table of Contents


• Dr. Koike requested that WTF-CEOP should include a data discovery function for CEOP data, using CEOP metadata.– JAXA agreed to develop a prototype CEOP data d

iscovery system under GCMD as a first step.– Lola established a CEOP portal in GCMD.– JAXA/RESTEC prepared samples of DIFs, based

on CEOP 19115 metadata samples and DIFs were ingested into CEOP portal.

CEOP InternationalMetadata Meeting


CEOP GCMD Portal


CEOP GCMD Portal

In-Situ

Surface datasets (EOP-3)

Tibet Himalayas Mongolia

Satellite

Subset datasets (EOP-1)

Aqua/AMSR-E

(Tibet)

ADEOS-II/GLI

(Inner Mongolia)

GCMD CEOP Portal:5 New DIF Samples

GCMD CEOP Portal:Data Discovery


CEOP GCMD Portal

• Work this year– Modify existing DIFs to provide a clear path to acc

ess WTF-CEOP from DIFs– Create DIFs for MOLTS data– Create DIFs for Model Output data









Table of Contents


Documentation

• Revise and expand User’s Manual

• Write System Maintenance Manual

• Recently updated WTF-CEOP Project Plan– Added information R&D plans for 3 years (to be

presented by Osamu)









Table of Contents


Implementation Plan #1 will finish at the end of this Fiscal year (March 31, 2006)

All 4 types of CEOP data will be fully functional

A new R&D effort is beginning this year and will continue for two more years (until March 31, 2008)

To be presented by Osamu

(c) Prof. Shibasaki@Univ. Tokyo

Conclusions andFuture plan

ceos/wgiss 20, kyev, september 12-16, 2005 1 wtf-ceop implementation plan #1 status (wtf-ceop first...

Documents

ceop implementation

ceop data policy slide

new data

data server software

data plots

data functions plot

various ceop data types

ceop portal