Common Data Index CDI V1How to proceed

ByDick M.A. Schaap – technical coordinator

Madrid, March 09

Central and homogeneous metadatabase plus transparent access to the distributed data sets via a unique user interface at the SeaDataNet portal and downloading services

SeaDataNet Delivery Services – CDI V1

SeaDataNet CDI V1 system components

CDI V1 Metadatabase and User Interface:Searching and browsing of metadata of data sets, managed by SeaDataNet Data Centres

Central User Register - Web service:Details of users, their organizations and addresses, license period, Id-Passwords, SeaDataNet Roles

Shopping Basket – User Interface:Preparing a user request of multiple data sets, handling validation of users and roles via the Central User Register, and routing requests to the Request Status Manager

Request Status Manager – Web service + User Interface:

Processing and administration of all requests and data deliveries (downloads), for users in communication with data centres. For data centres to oversee all transactions.

SeaDataNet CDI V1 system components

Data Centre Download Manager – Java Tool:Handles communication with Request Status Manager and takes care that requested files are made ready for downloading by User (if OK) in the agreed formats via a local website address at the Data Centre

SeaDataNet CDI V1 – implementation step by step

Prepare CDI V1 XML test records using the new MIKADO software and the latest CDI XML schema

Consider carefully the Data Access Restriction, that you want to apply to the data

Send the test records to the XML Validation Services for validation

If ok, then forward the test records to MARIS for inclusion in the CDI V1 test database

SeaDataNet CDI V1 – implementation step by step

Install the Download Manager software; 2 possible situations

You have a file management systemYou have a database management system

In case of file management system: Convert data files to SeaDataNet standard formats

Fill the ‘coupling table’ to connect the CDI V1 records (via LOCAL_CDI_ID) to the associated files (by name OR by query)

Test together with MARIS the set-up and good functioning of the connection via the Test interface

SeaDataNet CDI V1 – implementation step by step

If ok, then make full production of CDI V1 records, complete coupling table and make sure that the files are ready / the queries are ok.

Possible interim step: full CDI V1 production in the Import system for checking the quality of the CDI V1 records

Final step: MARIS moves the production CDI V1 records to the operational CDI V1 system and the data centre is active.

Maintenance and updating; only modifications => LOCAL_CDI_ID is important key to recognise updated or new CDI records

CDI is a fine-grained index to individual data measurements (such as a CTD cast or moored instrument record). This is in contrast to EDMED, which is an inventory of data sets, each of which comprises a large number of measurements.

CDI XML Schema is based upon the ISO 19115 standard

At the SeaDataNet website you can find:CDI V1 XML SchemaCDI V1 XML exampleCDI V1 logical description in XLS file

CDI V1 XML files can be prepared with the MIKADO software

SeaDataNet Prepare CDI V1 XML test records

SeaDataNet MIKADO Java tool

Available under multiple environments :Microsoft : Windows 2000, XP, VISTAAPPLEUnix - SolarisLinux

MIKADOJava code

Native DriversMYSQL

ORACLEPOSTGRESSQLServer

Bridge Driversusing Microsoft ODBC

(ACCESS, EXCEL, SQL SERVER)

DATABASE

JDBCJava DataBase Connectivity

EXCEL File

XML filesfor SeaDataNet

catalogues

CSREDMEDEDMERPCDI[EDIOS]

Manual

Automatic

Other DriversDownloaded from ad hoc Websites (Copied in the dist/lib MIKADO directory)

User must register once in order to get a personal login - passwordWeb form to provide necessary information User agrees with “SeaDataNet User Licence”After processing, login/password sent by email (email check)

Licence is part of the SeaDataNet Data Policy

NODCs intermediate in the registration of their national users to the SeaDataNet central user register => users will get a user role

Role together with the data access restriction determines the access:

User will have immediate access to the dataThe User request will be considered by the Data Centre and possibly the user will be contactedUser will be denied access

SeaDataNet – Consider the Data Access Restriction

User Roles in AAA Register, of relevance for data access

SDRN02 Public

SDRN03 Academic

SDRN04 Commercial

SDRN05 National and local government

SDRN06 Pan-National government

See Vocabulary for definitions Multiple roles may apply

Data access restrictions in CDI V1

Collection cost charge

Commercial charge

Licence

Moratorium

No access

Distribution cost

Organisation

Restricted

Academic

SeaDataNet licence

Unrestricted

See Vocabulary L081 for definitions Recommended to use: SeaDataNet

Licence!

Business Matrix

Restriction

Role

A B C D

U

V

W

OK

OK

OK

TBDTBD

TBDTBD

TBD

NO

NO

NONO

Recommended to use: SeaDataNet Licence!

SeaDataNet – Test CDI V1 records imported

Send the test records to the XML Validation Services for validation

http://www.seadatanet.org/validator

If ok, then forward the test records to MARIS for inclusion in the CDI V1 test database

http://seadatanet.maris2.nl/v_cdi_v1_test/search.asphttp://seadatanet.maris2.nl/v_cdi_v1_test/browse_step.asp

SeaDataNet CDI V1 – Data Access example

Search

Results

Include inBasket

Shopping list

Submit + Authentication

RequestConfirmed

Data

Check StatusIn RSM

Ready at DC x

Download

SDN format

SeaDataNet Download Manager V1.1 InstallationInstall and configure the Download Manager V1.1 software, following the installation instructions and in discussion with MARIS, who has to register the new data centre in the RSM configuration.

At the SeaDataNet Extranet (BSCW) in the Technical Task Team folders you can find all the relevant documentation and tools for the upgrade:

CDI V0 – V1 Steps guidance document (see Delivery Services folder)Specifications of the Download Manager (see Delivery Services folder)Technical documentation of the Download Manager (see Delivery Services folder)Installation instruction of the Download Manager (see Delivery Services folder)Software package of the Download Manager V1.1d (see Delivery Services folder)Transition document if you go from V1.1c to V1.1d (see Delivery Services folder)

SeaDataNet Download Manager V1.1 Installation

Install the Download Manager software; 2 possible situations

You have a file management systemYou have a database management system

In case of file management system: You have to convert data files to SeaDataNet standard formats

In case of database management system: Conversion is done ‘on the fly’ by the Download Manager software

Fill the ‘coupling table’ to connect the CDI V1 records (via LOCAL_CDI_ID) to the associated files (by name OR by query)

Test together with MARIS the set-up and good functioning of the connection via the Test interface

SeaDataNet Download Manager V1.1 FunctionsCommunication with RSM Web service about process progressProcessing order lists => retrieving requested data filesRetrieving data sets from:

file management systemsdatabase systems (e.g. Oracle, MySQL, MS-SQL, Sybase, ….)

If data sets are retrieved from a database => export of data files in SDN ODV4 formatNote: Data sets from a file management system must be pre-processed in SDN transport formats (ODV4 and optional MedATLAS for ASCII type data, or NetCDF (CF) for grid type data)

Combining data set files in daily zip files in a user download directoryGenerating dynamic user download webpage, following SDN template, to give user overview and option of downloadable zip files

SeaDataNet – Converting to SDN Data Formats

V1: data sets are accessable by downloading services. Delivery of data sets to users requires common data transport formats, which interact with other SeaDataNet standards (Vocabularies, Quality Flag Scale) and analysis & presentation tools (ODV, DIVA)

The following formats have been defined:SeaDataNet ODV4 ASCII for profiles, time series and trajectoriesSeaDataNet MedAtlas as optional extra format.NetCDF with CF compliance for gridded data sets

ODV4 and MedAtlas have been outfitted with a SeaDataNet semantic header

NEMO tool ready for conversion of ASCII files to SeaDataNet ODV and MedATLAS

Med2MedSDN V1.0 ready for conversion of MEDATLAS files without the SeaDataNet extensions to MEDATLAS files with SeaDataNet extensions (SeaDataNet MEDATLAS)

SeaDataNet CDI V1 – next steps

If the DM – RSM connection works fine in the Test system, then you are ready to move to full production and operational status

Make full production of CDI V1 recordsComplete the coupling tableMake sure:

associated files are ready OR

all SQL queries are ok.

Possible interim step: full CDI V1 production in the Import system for checking the quality of the CDI V1 records

http://seadatanet.maris2.nl/v_cdi_v1_import/search.asphttp://seadatanet.maris2.nl/v_cdi_v1_import/browse_step.asp

SeaDataNet CDI V1 – final steps

Final step: MARIS moves the production CDI V1 records to the operational CDI V1 system and the data centre is active.

http://seadatanet.maris2.nl/v_cdi_v1/search.asphttp://seadatanet.maris2.nl/v_cdi_v1/browse.step

Maintenance and updating; only modifications => LOCAL_CDI_ID is important key to recognise updated or new CDI records

SeaDataNet CDI V1 – Quick Search interface

Using an intuitive approach which is easy for users

SeaDataNet CDI V1 interim as temporary alternative

All Data Centres must populate the CDI V1 database:Preparing CDI V1 XML records following the upgraded CDI XML schema and using the new MIKADO software

V1 interim = no installation of Download Manager, but data centres will be informed by the Request Status Manager by e-mail of all requests

Data centres have to process all requests manually using the Request Status Manager

Delivering data sets in SeaDataNet Formats (conversion via new NEMO tool) – pre-processing required.

Effect is, that users will experience a common approach to all data centres. The difference is behind the scenes.

Thereafter centres should move to full V1 status by installing DM