![Page 1: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/1.jpg)
1
5a. SDMX and reference metadata exchanges
Bogdan ZDRENTUEurostatUnit B5: “Central data and metadata services”
SDMX Basics course, 27-29 October 2015
![Page 2: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/2.jpg)
EurostatEurostat
Structural metadata•acting as identifiers and descriptors of the data, such as:
• dimensions of statistical cubes• variables• titles of tables• Nomenclatures (code lists)
•always be associated with the data to allow their identification, retrieval and browsing.
Types of metadata
2
![Page 3: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/3.jpg)
EurostatEurostat
Example for structural metadata
3
![Page 4: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/4.jpg)
EurostatEurostat
Reference metadata•acting only as descriptors of the data, they don’t help to actually identify the data.•They can be of different kinds:
• conceptual metadata• methodological metadata• quality metadata (process and output)
•can be exchanged independently from the data they are related to, but are however often linked to them.
Types of metadata
4
![Page 5: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/5.jpg)
EurostatEurostat
Example for reference metadata
5
![Page 6: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/6.jpg)
ESS standardisation of reference metadata based on SDMX
6
The SDMX Content Oriented Guidelines (2009), i.e. the Cross-Domain Concepts and the MCV;
The SDMX Technical Standards: The information model for creation of the Metadata Structure
Definitions (MSDs);
The SDMX-ML for documenting the XML format;
The Euro SDMX Registry for storing the MSDs etc.
![Page 7: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/7.jpg)
EurostatEurostat
• Code lists describe dimensions in data tables, giving a meaning to the data.
• Code lists are based on:
• official statistical classifications such as NACE, NUTS, ISCO…
• the SDMX Content Oriented Guidelines
• Domain specific codifications
• A standard code list is a code list already harmonised
• Standard code lists should be used all along the statistical business process: data design, collection, aggregation, dissemination, archiving…
Standardisation of structural metadata
7
![Page 8: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/8.jpg)
EurostatEurostat
Example of a harmonised code list (NACE Rev. 1.1)
Old version (before harmonisation)
New version(after harmonisation)
Domains Old codes Old label_en New codes New label_en
hrst, htec MA_TOTAL Manufacturing sector
D Manufacturing
fats MAN Manufacturing industries
theme3 RD Manufacturing industry
theme4 B0200 Manufacturing industry
theme8 SE0_4 Manufacturing industry
theme9 TOT_MANUF Manufacturing industry
ds, hrst, htec MA_LOW_TEC Low technology manufacturing sector
D_LTCLow-technology manufacturingfats / inn LOT
Low Technology (incl. following NACE codes: 15-22; 36, 37)
inn I_LOW_TECLow tech industries: NACE Rev.1 codes 15 to 22, 36 and 37
hrst, htec SE_TOTALServices: NACE Rev. 1.1 sections G to Q = 50 to 99 G-Q Services
fats SER Services sector 8
![Page 9: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/9.jpg)
EurostatEurostat
• Better comparability: same codes for the same concepts
• Increase efficiency: less transcoding; less code lists; clean lists
• Improve accuracy: facilitate data management and exchange and reduce the number of errors
• Re-usability and integration of the data: data warehouse are only possible if codes corresponding to the same concept are the same
• SDMX implementation: it is essential for the implementation of a SDMX data/metadata exchange process.
• The ESS standard code lists will also be made available in the Euro SDMX Registry (currently RAMON).
Impact on the statistical business processes
9
![Page 10: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/10.jpg)
EurostatEurostat
RAMON
http://ec.europa.eu/eurostat/ramon10
![Page 11: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/11.jpg)
EurostatEurostat
Standard Code Lists in RAMON
11
![Page 12: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/12.jpg)
EurostatEurostat
The ESS Reference Metadata Standards
ESMS• Euro SDMX
Metadata Structure
ESQRS
• ESS Standard for Quality Reports Structure
EPMS
• Eurostat Process Metadata Structure
12
![Page 13: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/13.jpg)
EurostatEurostat
The Euro SDMX Metadata Structure (ESMS)
13
![Page 14: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/14.jpg)
EurostatEurostat
The ESS Standard for Quality Reports Structure (ESQRS)
14
![Page 15: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/15.jpg)
EurostatEurostat
Concept name 1 Contact
1.1 Contact organisation1.2 Contact organisation unit 1.3 Contact name 1.4 Contact person function 1.5 Contact mail address 1.6 Contact email address 1.7 Contact phone number 1.8 Contact fax number
2 Summary process description3 Workflow4 Statistical processing
4.1 Data collection 4.2 Source data
4.2.1 Source data - integration4.2.2 Source data - coding
4.3 Data validation 4.3.1 Data validation in Member States
4.3.2 Validation rules agreed with Member States4.3.3 Data validation - detection (Eurostat)4.3.4 Data validation - correction (Eurostat)
Concept name4.4 Data compilation
4.4.1 Data compilation - variables 4.4.2 Data compilation - weights 4.4.3 Data compilation - aggregates 4.4.4 Data compilation - finalisation 4.4.5 Data compilation - draftoutput
4.5 Data validation - final 4.5.1 Data validation final - output 4.5.2 Data validation final - explanation
5 Confidentiality 5.1 Confidentiality - data treatment
6 Release policy 6.1 User access
7 Dissemination format 7.1 Publications 7.2 On-line database 7.3 Micro-data access 7.4 Other
8 IT applications8.1 IT applications for data reception/collection 8.2 IT applications for data processing8.3 IT applications for data validation8.4 IT applications for data confidentiality8.5 IT applications for metadata8.6 Other IT applications
The Euro Process Metadata Structure (EPMS)
15
![Page 16: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/16.jpg)
EurostatEurostat
Dissemination of reference metadata
16
![Page 17: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/17.jpg)
EurostatEurostat
Dissemination of national reference metadata
17
![Page 18: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/18.jpg)
EurostatEurostat
The ESS Metadata Handler
Common user Interface
Output produced for the Eurostat Web
Other output for Eurostat or external users
ESS-MH IT application
RAMON CODED
ESS – Metadata Handler
Eu
ro S
DM
X
Reg
istr
yInput from national
metadataInput from national
metadataMetadata from the
Eurostat Domain managerMetadata from the
Eurostat Domain managerEurostat as main
administratorEurostat as main
administrator
The business process
18
![Page 19: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/19.jpg)
What is the ESS Metadata Handler?
19
The ESS Metadata Handler is a web based application for reference metadata production, exchange and dissemination in the ESS;
It implements the ESS metadata standards (ESMS, ESQRS and EPMS, etc.)
It replaces EMIS (used in Eurostat) and NRME (for countries);
It contains many improvements based on users' feedback (in terms of business process and functionalities);
It is in production since 31 January 2014.
![Page 20: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/20.jpg)
EDAMIS
National Statistical Institute
EUROSTAT
ESS Metadata Handler (ESS MH)
ESS MH Database Eurostat Website
PRODUCTIONTREATMENT & ANALYSIS
DISSEMINATION
National Metadata File
National Metadata File
National Metadata File
National Metadata File
National Metadata File
National Metadata File
20
![Page 21: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/21.jpg)
The business process for using the ESS MH for national metadata
Mapping of the existing national reference metadata files to the ESMS and/or ESQRS formats;
Conversion of existing national reference metadata files into standard structure;
Insertion of these files into the ESS MH application; The NSI’s are asked to complete, enhance their
converted files, directly in the ESS MH; The responsible Domain Managers in Eurostat are
asked to validate these ESMS / ESQRS files; The national metadata are finally disseminated on
Eurostat Web site (if decided so). Time line is approximately 6 months.
21
![Page 22: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/22.jpg)
ESS standards for metadata - Implementation
Waste (end of life vehicles, packaging, electronic waste)
WINE
FARM STRUCTURE
MIP STATISTICS
HICP/ Compliance monitoringEHIS (Education, health and social protection)R&D (CIS 2012)
Annual crops
PRAG ESAW AES (Education,
Science and Culture)
LCI (Labour Cost Index)
INFOSOC (Information Society)
BUSINESS REGISTER
HICP
LFS-Q, LFS-A
EU-SILC
FATS
STS (Short Term Statistics)
WASTE
AEI (Pesticides)
EDUCAT
JVC (Job Vacancy Stats)
PRODCOM
EXTERNAL TRADE (3rd countries)
COSAEA
URBANREG
R&D
TOURISM
PERMANENT CROPS
CENSUS
HOUSING PRICES HPS
22
![Page 23: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/23.jpg)
Practical Example
23
![Page 24: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/24.jpg)
Example for reference metadata
That source contains
metadata about the Tourism
datasets
24
![Page 25: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/25.jpg)
Let's select a specific topic from which we'll create a
metadata report
25 25
![Page 26: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/26.jpg)
Content of the selected topic
26
![Page 27: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/27.jpg)
Content of the selected topic
27 27
![Page 28: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/28.jpg)
Defining a Metadata Structure Definition
• The Tasks 1. Analysis of the entire set of metadata in order to
identify and document the “Concepts” for which metadata are to be reported or disseminated.
2. Determine the structure of the “Metadata Report” in terms of the concepts used, the hierarchy of the concepts when used in the report, and their “representation” (e.g. is a code list used, is the format free text?).
3. Specify the “object type” to which the metadata are to be attached, and how this object type is identified: knowledge of the SDMX Information model is useful here (as the metadata can only be attached to object types that can be identified in terms of the object types that exist in the information model). 28
![Page 29: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/29.jpg)
Metadata Report Structure – Content Metadata
BASIC_METH_ISSUESBASIC_METH_ISSUES
POS_ACC_TOURPOS_ACC_TOUR
STAT_UNITSTAT_UNIT
29
![Page 30: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/30.jpg)
SCOPE_OBSSCOPE_OBS
TOUR_ACC_ESTABTOUR_ACC_ESTAB
NACE_55_1NACE_55_1
30 30
![Page 31: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/31.jpg)
Metadata Report Structure – Concept SchemeThe following concepts are derived from this example:
BASIC_METH_ISSUES
POS_ACC_TOUR
STAT_UNIT
SCOPE_OBS
TOUR_ACC_ESTAB
CONTACT_ORG
CONTACT_ORG_UNIT
CONTACT_MAIL_ADDRESS
CONTACT
NACE_55_1 31
![Page 32: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/32.jpg)
Metadata Report Structure – Bringing it Together
Report Structure - Contact Report
CONTACT_ORG
CONTACT_ORG_UNIT
CONTACT_MAIL_ADDRESS
ESTAT_MSD
ESTAT_METADATA_CS
CATEGORY_REPORT
CONTACT
32
![Page 33: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/33.jpg)
METADATA REPORT STRUCTURE – BRINGING IT TOGETHER
Report Structure - Quality Report
ESTAT_MSD
ESTAT_METADATA_CS
CATEGORY_REPORT
BASIC_METH_ISSUES
POS_ACC_TOUR
STAT_UNIT
SCOPE_OBS
TOUR_ACC_ESTAB
NACE_55_1
33
![Page 34: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/34.jpg)
Metadata Set: Structure
• References to :• a Metadata Structure Definition (MSD)• a Report Structure• a Target Identifier
• Defines:• The actual values of the target objects
• Comprises:• The Reported Attributes and their corresponding
Values• These Attributes may be:
codedtextdate/time number etc.
34
![Page 35: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/35.jpg)
Metadata Set – General Schematic
CONTACT_ORG
CONTACT_ORG_UNIT
CONTACT_MAIL_ADDRESS
CONTACT
Unit G3 Short-term statistics; tourism
Eurostat, Statistical Office of the European Communities
http://epp.eurostat.ec.europa.eu/portal/page/portal/help/user_support
CATEGORY_REPORT
BASIC_METH_ISSUES
POS_ACC_TOUR
35
![Page 36: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/36.jpg)
METADATA SET – GENERAL SCHEMATIC
CATEGORY_REPORT
STAT_UNIT
SCOPE_OBS
36
![Page 37: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/37.jpg)
METADATA SET – GENERAL SCHEMATIC
CATEGORY_REPORT
TOUR_ACC_ESTAB
NACE_55_1
37
![Page 38: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/38.jpg)
METADATA SET – METADATA FILE
38
![Page 39: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/39.jpg)
METADATA SET – ESMS EXAMPLE
39 39
![Page 40: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/40.jpg)
METADATA SET – ESMS EXAMPLE
40 40
![Page 41: 1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015](https://reader035.vdocuments.us/reader035/viewer/2022062222/5697bfe81a28abf838cb6466/html5/thumbnails/41.jpg)
Questions?