page 1 envisat validation workshop, campaign database, 12/12/2002 envisat validation workshop...

Envisat Validation Workshop, Campaign Database, 12/12/2002

Envisat Validation Workshop

Atmospheric Chemistry Validation Team

Ground-Based Measurements and Campaign Database Subgroup

Campaign DatabaseTerje.Krognes, [email protected]

Aasmund.Fahre.Vik, [email protected] Norwegian Institute for Air Research (NILU)


PresentationPresentation OutlineOutline

Data Centre StatusTools

Local end-user tools

WWW search tools

Commissioning Phase Activities


Data flowData flow Signed protocol required

for access to tools, docs and files

ECMWF data requires additional protocolASC2HDF

SEARCH

HDF2ASC

OK?

DATA

DATABASE

table.dat

DOCSDATA ORIGINATOR

DATA USER

FILES

nadir web

ftp to zardoz

http

Error email

http or ftp


Data Centre Status (1/3): HardwareData Centre Status (1/3): HardwareThe machines have been swapped

between production and spare serversThe hardware has not been changed

Firewallzardozftp,

telnet

http

Storage

InternetInternetspare

nadir


Data Centre Status (2/3): SoftwareData Centre Status (2/3): Software Web-server (nadir) runs Linux Redhat 7.2 with

up2date ~weekly (6.2 at 2nd rehearsal) File server (zardoz) runs Solaris 7, as before Nusphere MySQL

Apache 1.3.22 (1.3.14) MySQL 3.23.44 (3.23.32) PHP 4.0.6 (4.0.4pl1)

IDL 5.4, as before HDF 4.1r3, as before No re-boot on nadir since April 2002 Uptime on zardoz since April 2002 is approximately

99,98% (2 half-hour maintenance + re-boot)


Data Centre Status (3/3): MetadataData Centre Status (3/3): Metadata

Metadata Guidelines updated by metadata board on 24 April 2002

During commissioning phase, updates of various metadata table entries, without affecting structure: Project-related information

Actual descriptors

Guidelines still applicable, but more clarifications to be added.


New user toolsNew user tools Major re-write of the ASC2HDF file generation and

checking tool around launch 6 smaller upgrades of the ASC2HDF package after

launch Entire ASC2HDF source code (FORTRAN) published to

users. HDF output routines may be combined with user’s own input and processing modules

IDL-tool to generate HDF data files (without error checking): IDL2HDF

IDL tool for reading and plotting HDF data files (HDF2IDL), to be downloaded and used locally

IDL subroutine to read HDF Cal/Val data, for integration into user-developed software (nilu_hdf_read_user.pro)


New web functionality New web functionality - ECMWF- ECMWF

ECMWF plots (NH 2,5º*2,5º and T106)

ECMWF 10-day back trajectory plots

T106 extractions to HDF files


New web functionality - search and plotNew web functionality - search and plot

Use of defaultsFacility to save and retrieve a searchProvision for downloading multiple files in tarball via web

interface Improved on-line plotting (also for multidimensional data)Corrected display of comments to variables


File searching (1/4)File searching (1/4)

Defaults are marked with *** Page initially opens without

defaults Page may be reset or set to

saved defaults bys selecting a previously saved query

Files may be filtered by a large number of metadata parameters



The default criteria page lists all criteria sets that this user has previously saved

Entries may easily be removed The main search page will be

loaded with the selected defaults when launched from this page

Defaults may include 4D space-time parameters and variable name wild-cards



The NUMERICAL SIMULATION entry is now marked with *** since we selected it as default through the saved query page

Variable name / mode / descriptor have wildcard entries automatically added to the drop down lists.

Variable name and variable mode accept multiple selections

4D space-time criteria may be entered as a single point, or as a max/min box

All files that include or touch the 4D criteria are listed as matches



Files may be selected by posting date

Files updated after a given date may be selected

The file list may be sorted by file name, by submitter or PI, by posting date, or by AOID

If no criteria are selected, the entire file list will be requested


File ListFile List The file list displays the current

selection criteria A single file may be downloaded

immediately Files may be marked and

downloaded in a compressed tarball

Search criteria may be saved for later re-use (type a meaningful description before saving)

The Variables link brings up a variable list and other documentation contained in the file


Variable ListVariable List The variable list page displays all

parameters contained in a file Variable name, mode and descriptor are

displayed The measurement unit and

corresponding SI unit are displayed All documentation fields attached to the

file are displayed Additional documentation fields attached

to a single variable may be displayed through the Comments link

Variables may be selected to generate an on-line plot (next slide)

Observe the dependencies listed for each variable – not all variable combinations will produce meaningful plots


Variable plotting (on-line browse)Variable plotting (on-line browse)


Administrative toolsAdministrative tools

Web tools to report uploads/downloadsWeb tools to archive/retrieve outdated

data filesWeb tools to check and maintain index

database consistency and integrity


User activitiesUser activities Users have sent approximately 200 emails with requests for accounts

or updates to metadata, or with questions to HDF file generation After launch focus has slowly shifted to requests for help with HDF file

formatting issues and removal of outdated files From May to November 2002 users have submitted 5880 HDF data

files (570 of these were rejected due to formatting problems, and 238 have later been archived).

From May to November 2002 users have downloaded 4000 HDF data files (totalling 215 MB) (ftp downloads via zardoz not included)

There are currently 4741 ACVT files open to users (22 active AOIDs) There are currently 462 MAVT files open to users (5 active AOIDs)


User activities: Up and DownloadsUser activities: Up and Downloads

0

200

400

600

800

1000

1200

35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

uploads

downloads

The download statistics only covers WWW downloads from nadir, not FTP downloads from zardoz


Nadir-team activitiesNadir-team activities 300+ interactions with users (of which 50% during Commissioning Phase) 250+ interactions with ESA and the Metadata Board (10% during

Commissioning Phase) One major re-write and 6 updates of the ASC2HDF and HDF2ASC programs Maintained and improved shell scripts for automatic file checking and storage Created additional IDL and FORTRAN tools Performed extensive web program development Performed user administration updates Added metadata entries and updated the table.dat file 23 times since launch Developed and maintained scripts and data flow to provide pre-processed

ECMWF data extractions and plots on the web Monitored the automatic file reception system (now containing nearly 5000 files

totalling 275 MB + index database 20 MB)


Metadata issuesMetadata issuesAlthough many quality checks are performed on the data upon submission to

the database: The metadata definition is currently not complete enough to perform

automatic checks of the validity of reported measurement units Some components and units may still be inconsistently named or mis-used

Several files have indeed been submitted containing such errors due to misinterpretation of the guidelines. The data submittors will be contacted to correct these files.

page 1 envisat validation workshop, campaign database, 12/12/2002 envisat validation workshop...

Documents

data database

files ecmwf data

data flow

hdf calval data

ftp slide

hdf files

new web functionality

multiple files