reformatting tools med2medsdn and nemo
Post on 11-Jan-2016
41 Views
Preview:
DESCRIPTION
TRANSCRIPT
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
REFORMATTING TOOLS
Med2MedSDN and NEMO
M. Fichaut
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
2www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
CDI
CSR
EDMERP
EDMED
SEADATANET PORTAL
European portal
Collection of
ASCII filesFormat X
Data in Database
Metadata in
Database
Data Input
Metadata Input
MetadataIn
Excel files
Partner system : pilot data centre
SeaDatanet Vocabulary
NEMO
ODV
Med2MedSDN
Coupling table
Download
Manager
Collection of
ASCII filesFormat SDN
XML Metadata
Files
Local copy of data
to download
Data request
Data download
MIKADO
MIKADO
XML Validator
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
3www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Overview
• Med2MedSDN
• Objectives and main features
• NEMO
• Objectives, main features and principles
• Technical characteristics
• Description of the different steps to follow to be able to reformat ASCII files to SeaDataNet format
• Link between NEMO and MIKADO
• Link between NEMO and SDN Download Manager – coupling table
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
Med2MedSDN
For MEDATLAS files
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
5www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Objectives of Med2MedSDN
• Translate MEDATLAS files to SeaDataNet MEDATLAS files - available under Windows
• Version 1.1.02 and user manual available at :
http://www.seadatanet.org/standards_software/software/Med2MedSDN
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
6www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Med2MedSDN main features
• Reformats MEDATLAS files to MEDATLAS SeaDataNet format
• Java tool, bilingual (French, English)
• Adds the additional SeaDataNet information : mapping for parameters and LOCAL_CDI_ID and EDMO_CODE
• Can reformat one file or a large number of files (in one directory)
• Generates a coupling file for the SeaDataNet download manager
• Linked to SeaDataNet vocabularies through Web services for parameters mapping and for list of EDMO codes
• Need of internet connexion while updating lists
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
7www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Med2MedSDN
Med2MedSDN vocabulary update
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
8www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Med2MedSDN main screen
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
9www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Med2MedSDN log files
• Errors (managed by Log4j) are registered in a log file in Med2MedSDN installation directory.
• One line in the log file is composed as following:• Date, Name of the Software, Error severity level, Error message
INFO Informative messages for starting of the conversion or successful conversion
ERROR For conversion errors : conversion is cancelled on the current file but continues on the other files
FATAL For conversion errors which stop the processing of the files
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
10www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Med2MedSDN - User manual
• User manual is provided :
File : sdn_Med2MedSDN_UserManual_V1.0.pdf
• Detailed explanation for Med2MedSDN installation and use, lots of snapshots
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
11www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
ANY QUESTION?
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
www.seadatanet.org
Training Workshop – Ostende – 26-29 October 2009
NEMO
For ASCII files on any type
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
13www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Objectives
NEMO is a reformatting software used to
generate ASCII files at MEDATLAS or ODV formats which are defined as SeaDataNet formats for data exchange between SeaDataNet partners. Can be downloaded from SeaDataNet Web site :
http://www.seadatanet.org/standards_software/software/nemo
It was created in IFREMER for internal purposes, and adapted to take into account the SeaDataNet formats and vocabularies.
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
14www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Technical characteristics• Written in Java Language (Version >= 1.6)
• Bilingual (French, English)
• Available under multiple environments :
• Microsoft : Windows 2000, XP, VISTA , APPLE
• Unix - Solaris
• Linux
• Use of SeaDataNet common vocabularies web services
• to update lists of values of the SeaDataNet common vocabularies
• need network connections in order to have up to date lists of values.
• But NEMO works offline once the lists are up-to-date
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
15www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
NEMO main features
• Reformat any ASCII file of vertical profiles, time-series or trajectories to a SeaDataNet ASCII format (ODV, MEDATLAS).
• The input ASCII files can be :
• one file per station for vertical profiles or time series
• one file for one cruise for vertical profiles, time series or trajectories
• Interact with Mikado, to be able to generate ISO-19115 XML descriptions of the data.
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
16www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
NEMO principles• NEMO must be able to read almost any ASCII format to
translate it to MEDATLAS, ODV• Users of NEMO describe the entry files format so that NEMO
is able to find the information which is necessary in the SeaDataNet formats
• One mandatory pre-requirement is that in the set of entry files the information about the stations must
• be located at the same position : same line in the file, same position on the line or same column if CSV format
• be in the same format• For example : for all the stations the latitude is
• on line 3 on the station header, • from character 21 to character 27, or 3rd column in CSV• the format is +DD.ddd
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
17www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Running NEMO
• 4 or 5 steps to convert the entry files to a SeaDataNet format1. Describe the type of file(s)2. [Describe the cruise, if the files are related to 1 cruise]
• NEMO is able to read XML CSR generated with MIKADO
3. Describe the station information4. Describe the measured parameters5. Convert the file
• One more step to • Save the description of the format (Model)
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
18www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
NEMO - Description of entry files (1)
• Where are the files?
• Is it a cruise?
• Is it one file per cruise or one file per station ?
• Is it a collection of station files ?
• grouped by cruises or not?
• Are the files with separators?
• Tabulations? Semicolon? Comma?
• Are they vertical profiles, time series or trajectories data?
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
19www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
NEMO - Description of entry files (2)
• File to convert is displayed in NEMO window
• The principle is to describe the input file by selecting the information in the file window
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
20www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Station header : constant number of lines
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
21www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Station header : number of lines not constant
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
22www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Cruise description
Can be
• Manually input
• Imported from a SeaDataNet XML CSR
• Imported from a database (IFREMER only)
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
23www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Station description
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
24www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Station description
• Select the date in the file, in the left part of the screen
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
25www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Station description
• Input the date format
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
26www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Station description
• Click on the Set button to tell NEMO the start and end position of the date
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
27www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Station description
• Click on the Test button to verify your input
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
28www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Station description
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
29www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Data description
• Give the position , the default value, the format of each parameter
• QC flags can also be described
• Pressure is mandatory for vertical profiles (MEDATLAS requirement) and cannot be deleted.
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
30www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Data description
• Select « Parameter list » in the context menu of the measured parameters table (right click to open the menu)
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
31www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Data description
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
32www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Data description
• Select the value of the measured parameter in the file, in the upper left part of the screen
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
33www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Data description
• Select « Set start /end » in the context menu of the measured parameters table
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
34www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Data description
• Start and End position of temperature are added in the table
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
35www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Data description
• Select « Update test » in the context menu of the measured parameters table
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
36www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Data description
• The values read by NEMO are displayed in the Test column of the table. User is able to check that his input is correct
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
37www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
File conversion
Output formats
• MEDATLAS
• Multi-station
• ODV
• Mono-station
• Multi-station
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
38www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
While converting …
NEMO is able to
• Generate a SeaDataNet CDI Summary
• Text file containing the minimum mandatory information needed in the CDI ISO-19115 description
• This file can be converted to an Excel file
• Which can be read by MIKADO to generate the XML CDI files
• Generate a coupling table that will be used by the Download manager of SeaDataNet
• It is the link between the LOCAL_CDI_ID and the file
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
39www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
CDI summary
• Further information must be added in NEMO settings (menu Options of NEMO)
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
40www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
CDI SUMMARY FileText file
• Open it with Excel
• Rename the sheet : STATION
• Save the file as an EXCEL file
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
41www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Interaction with MIKADO
Collection of
ASCII files
ODV files
CDI summaryCSV file
SeaDataNetCDI
MIKADO
XML CDI files
summary_CDI_NEMO.xml
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
42www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Coupling table for Download Manager
• Further information must be added in NEMO settings (menu Options of NEMO)
• The coupling table makes the link between each LOCAL_CDI_ID and the file(s) which contains it
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
43www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Coupling table
If the “Generate mapping” option is clicked :
• NEMO will insert a record in the coupling table for each vertical profile, time series or trajectory that it converts.
• The coupling table is managed by a local database imbedded in NEMO
• The coupling table can be :
• Edited (for modification, insertion or deletion)
• exported (to be used by the download manager)
• imported (from previous version of NEMO, for example)
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
44www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Coupling table content (1)
LOCAL_CDI_ID
Modus
• 1 : mono-station
• 3 multi-station
Filename
Date of creation
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
45www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Coupling table content (2)
• One unique LOCAL_CDI_ID in one mono-station ODV file
• The same LOCAL_CDI_ID in 2 different files :
• One multi-station MEDATLAS file
• One mono-station ODV file
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
46www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
Downloading the data
CDI
CSR
EDMERP
EDMED
SEADATANET PORTAL
European portal
Collection of
ASCII filesFormat SDN
Local copy of data
to downloadData request
Data download
Coupling table
SeaDataNet Data Centre
List of LOCAL_CDI_IDs
DownloadManager
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
47www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
NEMO – User manual
• User manual is provided :
File : sdn_Nemo_UserManual_V1.2.pdf
• Detailed explanation for NEMO installation and use, lots of snapshots
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
48www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
And don’t forget that
• All software freely available at :
http://www.seadatanet.org/standards_software/software
• FAQ on the web site for MIKADO and NEMO
• Don’t forget to read the user manuals which are very detailed
• Any problems using the software
Mail to sdn-userdesk@seadatanet.org
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
49www.seadatanet.org
3rd SeaDataNet training course – Ostende – 27 October 2009
ANY QUESTION?
top related