observations & prÉvisions cÔtiÈres 3 rd seadatanet training course – ostende – 16-19...
TRANSCRIPT
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
www.seadatanet.org
3rd SeaDataNet training course – Ostende – 16-19 June 2008
Tools for data management in the frame of SeaDataNet
project
M. Fichaut
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
2www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
CDI
CSR
EDMERP
EDMED
SEADATANET PORTAL
European portal
Collection of
ASCII filesFormat X
Data in Database
Metadata in
Database
Data Input
Metadata Input
MetadataIn
Excel files
Partner system : pilot data centre
SeaDatanet Vocabulary
NEMO
ODV
Med2MedSDN
Coupling table
Download
Manager
Collection of
ASCII filesFormat SDN
XML Metadata
Files
Local copy of data
to download
Data request
Data download
MIKADO
MIKADO
XML Validator
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
CDI
CSR
EDMERP
EDMED
SEADATANET PORTAL
European portal
Collection of
ASCII filesFormat X
Data in Database
Metadata in
Database
Data Input
Metadata Input
MetadataIn
Excel files
Partner system : other data centre
SeaDatanet Vocabulary
NEMO
ODV
Med2MedSDN
Collection of
ASCII filesFormat SDN
XML Metadata
Files
Data download
Local copy of data
to download
Data requestby email
Manual preparation of data
MIKADO
MIKADO
XML Validator
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
4www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
Overview (1)• MIKADO
• Objectives and technical characteristics of Mikado
• Mikado’s main features
• Manual input of metadata
• Use of the common SDN vocabularies
• Automatic XML generation
• Connection to the local database
• Query writing
• Mapping to the local database
• Batch mode
• Coupling table for download manager
• XML VALIDATION SERVICE
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
5www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
Overview (2)
• Med2MedSDN
• Objectives and main features
• NEMO
• Objectives, main features and principles of NEMO
• Technical characteristics
• Description of the different steps to follow to be able to reformat ASCII files to SeaDataNet format
• Link between NEMO and MIKADO
• Link between NEMO and SDN Download Manager – coupling table
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
www.seadatanet.org
3rd SeaDataNet training course – Ostende – 16-19 June 2008
MIKADO
Tool for the generation of XML descriptions of SeaDanaNet
catalogue records
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
7www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
Objective
MIKADO is used to generate XML catalogue descriptions, it creates XML ISO-19115 files using SDN common vocabularies for metadata exchange of
CSR - Cruise Summary Reports
EDMED - Marine Environmental Data sets
CDI - Common Data Index
EDMERP - Marine Environmental Research Projects
[EDIOS – Permanent Ocean-observing System]
Is freely available on SeaDataNet Web site
http://www.seadatanet.org/standards_software/software/mikado
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
8www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
Technical characteristics
• Written in Java Language (Version >= 1.6)
• Available under multiple environments :
• Microsoft : Windows 2000, XP, VISTA , APPLE
• Unix - Solaris
• Linux.
• Use of the SeaDataNet common vocabularies web services to update lists of values
• needs network connection in order to have up to date lists of values.
• but Mikado works offline once the lists are up-to-date
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
9www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO main features (1)MIKADO can be used in 2 different ways
• One manual way, to input manually information for the catalogues and CDI in order to generate XML files.
• One automatic way, to generate XML descriptions automatically, from information catalogued in a relational database or in an Excel file. Automatic way is needed for those who have many entries referenced in a relational database
Only one interface for all catalogues
• Same look
• Same principles
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
10www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO main features (2)
MIKADOJava code
Native DriversMYSQL
ORACLEPOSTGRES
SYBASEMSServer
Bridge Driversusing Microsoft ODBC
(ACCESS, EXCEL, SQL SERVER)
DATABASE
JDBCJava DataBase Connectivity
EXCEL File
XML filesfor SeaDataNet
catalogues
CSREDMEDEDMERPCDI[EDIOS]
Manual
Automatic
Other DriversDownloaded from ad hoc Websites (Copied in the dist/lib MIKADO directory)
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
11www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO and SDN vocabulary lists (2)
List code
List Name List code
List Name
C16 SeaDataNet Sea Areas L061 SeaDataNet Platform Classes
C77 ICES ROSCOP data types L071 SeaDataNet data access mechanisms
C174 SeaDataNet CSR ship metadata L081 SeaDataNet Data Access Restriction Policies
C320 ISO countries L101 SeaDataNet geographic co-ordinate reference frames
C371 Ten-degree Marsden Squares L111 Height and Depth Vertical Co-ordinate Reference Datum
C381 Ports Gazetteer L181 ROSCOP sample quantification units
L05 SeaDataNet device categories L231 SeaDataNet metadata entities
L021 SeaDataNet Geospatial Feature Types L241 SeaDataNet data transport formats
L031 SeaDataNet Measurement Periodicity Classes P021 BODC Parameter Discovery Vocabulary
EDMERP European marine projects P081 SeaDataNet Parameter Disciplines
EDMO European marine organisation
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
12www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO and SDN vocabulary lists (2)
•Automatic check of the version of the vocabulary lists : once when MIKADO starts
• If “On” is clicked in the Vocabulary Update Menu
• MIKADO downloads locally the latest version of each list
•Possible to enable-disable the automatic check
• If “Off” is clicked
•Manual check
• Update once now
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
13www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO – Manual input
• Available for 4 catalogues : EDMED, CSR, CDI, EDMERP
• Each input generates an XML file that can be sent to the central catalogue
• For EDMERP and CSR : EDMERP CMS and CSR online can also be used, but MIKADO is useful
• if you have problems with the NETWORK connection
• if you want to keep locally an XML description of your catalogues
• For EDMED and CDI, there is no online input tools.
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
14www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO – Manual input
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
15www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO manual : LOCAL Identifier
• The LOCAL Identifier is vital because it is kept in the central catalogue and is the entry point to know if the record is new or if it is an update.
• This LOCAL ID exists for all the catalogues and is under the responsibility of the data centre who generates the XLM descriptions
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
16www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO manual – vocabulary lists (1)
Common vocabulary BODC,
list C320 (Country ISO codes)
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
17www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO manual – vocabulary lists (2)
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
18www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO – Automatic XML generation
Principle
• Read the information about CSR, EDMED, EDMERP or CDI in a database or in an Excel file
• MIKADO has predefined variables which correspond to the XML tags definition for each catalogues
• MIKADO helps user to write the SQL orders to fulfill these variables with the information available in the database or in the Excel file
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
19www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO – Automatic XML generation
4 STEPS
• Connect to a database or an Excel file and test the connection
• Write the queries to retrieve information in the database or in the Excel file, test the queries
• Save the queries in a “Configuration file”
• Generate the XML files using the “Configuration file”
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
20www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO – Automatic XML generation
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
21www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO automatic - connection
• Help for the connection to the database
• Pre-filled information for some databases
• Check of the connection
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
22www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO automatic – connection OK
• Green message in the Check box
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
23www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO – automatic – connection KO
• Red message in the Check box
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
24www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
Expendable trees
• Main query
Return the LOCAL ID
• Single queries
Return 1 row
• Multiple queries
Return 1 to n rows
• Single an multiple queries related to each LOCAL ID returned by the main query.
MIKADO automatic – queries
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
25www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
Write the queries
• SQL syntax (for Oracle, Excel, MySQL, …) and SQL variables must be adapted to your own data base
Check the Queries
• Green OK
• Red KO : read the error message
MIKADO automatic – queries
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
26www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO automatic – single queries
All the XML variables are listed in the expendable tree
• In bold : mandatory fields
• 1 to n single subquery can be written
• In green : fields already fulfilled
• Add or delete variables in a query
• Delete a full query
• Check the query
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
27www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO automatic – multiple queries
All the XML variables are listed in the expendable tree
• Number of queries is pre-defined
• The list of variables for each of these multiple queries is also pre-defined
• In bold : mandatory field
• In green : fields already fulfilled
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
28www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO automatic – multiple queries
• In a group of variables (same XML block, same pre-defined set of variable in a query), the non mandatory variables can be left to null
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
29www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO automatic - Save the queries
When all the queries are written
• They can be saved in an XML file to be re-used later on
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
30www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO automatic - generate the XML files
1. Select the catalogue you want to generate
2. Open the corresponding configuration file
3. Choose the output directory
4. Choose the type of export files
5. Export the XML files
• Progress bar
• Cancel allowed
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
31www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
While generating the XML files for all the catalogues
• Each time that MIKADO does not recognized a value (entrykey or entryterm) which should come from the common vocabulary, it asks the user for mapping
• MIKADO manages a demand-driven continuous (incremental) extension of a local mapping : mapping of the local database to the common vocabulary
• Mapping tables can be modified
• Delete rows
• Modify the LOCAL value
MIKADO automatic - local mapping
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
32www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO automatic - local mapping
Example
CSR generation
• Mapping of the platform type
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
33www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO automatic - local mapping
Modification of the local mapping
If wrong entries have been input, it is possible to:
• Delete one entry
• Delete all the entries
• Change the local code
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
34www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO in batch mode
• MIKADO can be run in batch mode using existing configuration files
• Several arguments can be added on the command line
Java –Djava.endorsed.dirs=”dist/lib” –jar dist/Mikado.jar mikado-home=[path] argument2= … argumentn=
• Log file to register the errors
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
35www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
Batch mode : mandatory arguments (1)
batch-type : XmlFiles, Zipfile, Both
batch-mode : CDI, EDMERP, CSR, EDMED
conf-file : path and name of the configuration file with the SQL queries
output-dir : path and name of the output directory where the XML files will be written
continue-on-error : true or false
If true : if one record with mapping missing, or one record with mandatory field(s) null in the database, warning for this record in the logfile, and MIKADO processes next records..
If false : MIKADO stops at the first error.
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
36www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
Batch mode : optional arguments (2)
log-file : Path and name of the Log file of MIKADO. By default, mikado.log in Mikado home directory
trace : to have a summary of the time response of each SQL query (useful for tuning of the queries bad if time responses)
max-file-in-zip : number maximum of XML files per zip file generated by Mikado. By default, one zip file contains 1000 XML files.
zip-prefix : To personalize the names of the Zip files generated by Mikado. By default the zip file name are SeaDataNet_[catalogue]_[x].zip
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
37www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
Coupling file for Download manager (1)
• MIKADO is able to generate this coupling file
• The coupling file is used by SeaDataNet download manager to make the mapping between a LOCAL_CDI_ID (one profile, one time-series or one trajectory) and
• the name of the file containing this LOCAL_CDI_ID (MODUS1 and 3) : if the metadata is in a data base and the data in files
• the SQL Query to retrieve the meta data and the data of this LOCAL_CDI_ID in the local database (MODUS 2)
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
38www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
Coupling file for Download manager (2)
• The principle to create this coupling file is the same than to create XML files for catalogue descriptions
• User has
• to create a configuration file that will be used for the generation of the coupling file.
• to write the queries to retrieve the filename or the data for each LOCAl_CDI_ID
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
39www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
Coupling file for Download manager (3)
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
40www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
MIKADO – User manual
User manual is provided :
File : SDN_MIKADO_UserManual_v1_1_4.pdf
• Detailed explanation for MIKADO use, lots of snapshots
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
41www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
ANY QUESTION?
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
42www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
VALIDATION of XML files
• XML validation Services have been developed in the frame of SeaDataNet by the Russian NODC
• It is a Web validation Service available at
http://www.seadatanet.org/validator
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
www.seadatanet.org
3rd SeaDataNet training course – Ostende – 16-19 June 2008
XML validation tool
Web service for validation of the XML files generated by MIKADO
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
44www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
• Select the catalogue you want to validate
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
45www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
• View all CDI schema versions
• View statistics about detected errors
• Run the validation with the last version of XML schema
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
46www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
• Upload the file you want to validate
• Validation works only file per file
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
47www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
• The file is displayed on the screen
• Press Validate
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
48www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
• Errors and warning are displayed
• Warning are not obstacles for XML delivery
• Errors must be corrected
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
49www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
ANY QUESTION?