dataset registration process sergey sukhonosov, dr. sergey belov national oceanographic data centre,...
TRANSCRIPT
![Page 1: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/1.jpg)
Dataset registration process
Sergey Sukhonosov, Dr. Sergey BelovNational Oceanographic Data Centre, Russia
Training course on establishment of the ODP regional node and data network for SNDM-Argentina,7 - 11 October 2013, Buenos Aires, Argentina
![Page 2: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/2.jpg)
2
Registration stepsDesignRegistrationMaintenance
![Page 3: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/3.jpg)
3
DesignData source storage type
DBMS – all types of data;Structured data files – forecasts, climate, small volumes of
real-time data;Objective files – images, documents, shape-files, …;Web-application – WMS;
Data source structure (data files, tables/views)
![Page 4: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/4.jpg)
4
Data source storage type - DBMSSupported databases: MySQL, PostgreSQL, Oracle, MS
SQL Server (native JDBC driver)Table structure requirement: one parameter – one
column.Possibility to get the data from one or several tablesFilter the dataBenefits:Metadata is updating automaticallyCaching modeFast processing of large amounts of data.Flexible adjustment of the data.
![Page 5: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/5.jpg)
5
Data source storage type – Structured data files
Supported formats: text files with separator or text files with fix column positions.
Structure must be ‘plain’, non-hierarchical
Options to link data files:Specify URL to HTTP/FTP server,Upload data file to Data Provider server using web
interface,Copy data file to local Data Provider file system
![Page 6: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/6.jpg)
6
Data source storage type – Structured data files
Benefits:• Metadata is updating automatically• Caching mode• In case of using HTTP/FTP data source it’s enough to
update data files, Data Provider will synchronize data, update data cache (instances) and update dynamic metadata (temporal extent, geographic bounding box,…)
• Connect set of data files to one metadata (specifying all file names or using filename mask (e.g. “*.txt”))
![Page 7: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/7.jpg)
7
Objective data filesSupported formats: any data format – Data Provider
don’t analyze the content and provide data “as is”. In case if several object data files should be linked to one
metadata, they can be zipped to archive. E.g. GIS data (shp, shx, dbf).
The ways to link objective data files:• specify URL to HTTP/FTP server,• upload data files to Data Provider server using web
interface
![Page 8: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/8.jpg)
8
Objective data filesAt least one object data file must be linked to discovery
metadata. Linked data files call ‘instances’ and have their own discovery metadata.
Metadata for instance based on resource metadata.Features:• Once data file is uploaded it can’t be updated
automatically: only manually.• You should add new data file manually via web-interface• Dynamic metadata is not updating automatically by
scheduler
![Page 9: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/9.jpg)
9
Design
Data granularity issue: a) Split data to several metadata records by geographic
box, time period, measured parameter;
![Page 10: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/10.jpg)
10
Designb) Split data to data subsets (instances) by “key”
parameter;
![Page 11: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/11.jpg)
11
Data granularity – point data
![Page 12: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/12.jpg)
12
Data granularity – point data
2001-08-06T00:00:00;-7.778;119.603;V2AP6.00;0;27.3
2001-08-06T00:00:00;-7.778;119.603;V2AP6.00;68;27.2
2001-08-06T00:00:00;-7.778;119.603;V2AP6.00;92;26.4
2001-08-06T00:00:00;-7.778;119.603;V2AP6.00;100;25.6
2001-08-06T00:00:00;-30.515;-155.227;DCFH2.00;2;17.9
2001-08-06T00:00:00;-30.515;-155.227;DCFH2.00;4;17.9
2001-08-06T00:00:00;-30.515;-155.227;DCFH2.00;6;17.9
2001-08-06T00:00:00;-30.515;-155.227;DCFH2.00;8;17.9
![Page 13: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/13.jpg)
13
Data granularity – profile data
![Page 14: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/14.jpg)
14
Data granularity – profile dataSpecifying record header
![Page 15: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/15.jpg)
15
Data granularity – profile dataSpecifying list of parameters in the data table
![Page 16: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/16.jpg)
16
Data granularity – profile data
2001-08-06T00:00:00;-7.778;119.603;V2AP6.000;27.368;27.292;26.4100;25.6
2001-08-06T00:00:00; -30.515;-155.227;DCFH2.002;17.94;17.96;17.98;17.9
![Page 17: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/17.jpg)
17
System parameters analysisMake cross-mapping between local data columns and
system parametersContact global node (MINCYT) to add new parameters if
required
![Page 18: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/18.jpg)
18
RegistrationUse web-interface to register metadata and connect the
data (Notice: Chrome browser is not supported yet)Use metadata templatesSpecify maximum details to fit metadata quality
requirements
![Page 19: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/19.jpg)
19
MaintenanceUpdate data in-timeUpdate metadata on data frequency basis – create
schedulers for automatic metadata actualizationAnalyze daily reports from Integration Server
![Page 20: Dataset registration process Sergey Sukhonosov, Dr. Sergey Belov National Oceanographic Data Centre, Russia Training course on establishment of the ODP](https://reader035.vdocuments.us/reader035/viewer/2022062805/5697bfc01a28abf838ca3b73/html5/thumbnails/20.jpg)
20
Questions?