![Page 1: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/1.jpg)
SDMX implementation: Status and next steps Item 3.2 – Standards Working Group 2-3 July 2018 Luca Gramaglia, Unit B5
![Page 2: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/2.jpg)
SDMX and its benefits
![Page 3: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/3.jpg)
Technical standards (Information Model and formats)
Statistical guidelines (common concepts and code lists)
Automation benefits (notably for validation)
Semantic harmonisation benefits
![Page 4: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/4.jpg)
Current status of SDMX implementation
![Page 5: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/5.jpg)
History
• In 2009, Eurostat started implement SDMX in all new or considerably changed datasets and reference metadata sets.
• Virtually all SDMX implementation projects carried out in Eurostat have sought to achieve compliance to both the SDMX technical standards and to the SDMX statistical guidelines.
![Page 6: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/6.jpg)
A look at EDAMIS
• To assess the status of SDMX implementation, the EDAMIS transmission logs from 2015 to 2017 were analysed.
• Focus was on official data transmissions (no metadata transmissions or transmissions for validation or testing purposes).
• Data transmissions from Member States and to/from international organisations were analysed
![Page 7: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/7.jpg)
Focusing on data transmissions…
National Authorities
Eurostat
International organisations
74515
11
11146
2398 2083
413
![Page 8: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/8.jpg)
Data provider categories
![Page 9: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/9.jpg)
% of SDMX transmissions by data provider category
![Page 10: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/10.jpg)
Preferred SDMX formats
![Page 11: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/11.jpg)
% of EDAMIS datasets that predominantly use SDMX formats
International data exchange
2015 2016 2017
EWF 0.0 0.0 0.0
GESMES 58.8 42.9 34.3
SDMX-ML 35.3 42.9 54.3
TOTAL
SDMX 94.1 85.7 88.6
![Page 12: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/12.jpg)
% of EDAMIS datasets that predominantly use SDMX formats
National data collection
2015 2016 2017
EWF 18.8 17.1 14.5
GESMES 6.3 6.1 5.6
SDMX-ML 12.2 14.0 14.1
TOTAL
SDMX 37.3 37.2 34.2
![Page 13: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/13.jpg)
Main conclusions
• For data exchange with international organisations, SDMX is the de facto standard
• For data collection from national data providers, sow or no growth over past three years
• Certain categories of national data providers (ONAs in particular) experience difficulties. SDMX implementation is successful with these data providers when the complexity of SDMX is hidden (e.g. when Webforms are used)
![Page 14: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/14.jpg)
Moving forward: Benefit-based implementation
packages
![Page 15: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/15.jpg)
• The difficulties in implementing SDMX can be summarised by the fact that the approach to implementation requires a lot of changes from business stakeholders, both in Eurostat and in Member States.
• Changes mostly come from: Changes are required to the structure of the data
Changes are required to the codes used
Changes are required in the format used
• The solution are different implementation packages that are adapted to specific business needs
Issues
![Page 16: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/16.jpg)
SDMX compliance – technical side
Definition Business benefits
Criterion 1
A DSD which describes the structure of the incoming data is available. The data is not necessarily in a standard SDMX format, but can be mapped to the DSD.
Structural validation can be performed based on the DSD. Content validation rules can be implemented based on the DSD.
Criterion 2
The data is in a standard SDMX format.
Once the DSD is available, no additional specifications are required to describe the format.
![Page 17: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/17.jpg)
SDMX compliance – statistical side
Definition Business benefits
Criterion 1
Concept names are aligned to the ones recommended in the SDMX Glossary.
Basic level of semantic interoperability between different datasets
Criterion 2
The guidelines for SDMX code lists are applied and cross-domain SDMX code lists are used.
Reduced need for transcoding when integrating or joining data across different domains.
Criterion 3
Compliance to the SDMX modelling guidelines (e.g. standard recommended concepts are used)
Data integration across different domains is easier, with little or no mapping between structures and codes necessary.
![Page 18: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/18.jpg)
Definition of the implementation packages
![Page 19: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/19.jpg)
SDMX-based validation for everyone The light implementation
• The light implementation focuses on making available SDMX-based validation tools to all statistical domains
• Does not foresee changes in structure/codes and does not prescribe switching to an SDMX standard format. Only seeks to describe the current structure of the data via a DSD to enable automated SDMX-based validation
• Effort is foreseen to be small for domains that currently receive data in CSV or FLR, larger for domains using Excel.
• Applicable to both aggregate data and micro-data.
![Page 20: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/20.jpg)
SDMX-based validation for everyone The light implementation
• Light implementation gives access to the validation architecture developed by the ESS Vision 2020 Validation project.
• Should represent the minimum SDMX implementation level for all domains
![Page 21: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/21.jpg)
SDMX-based content harmonisation The medium and full implementation
• Full implementation of the SDMX standard still available when content harmonisation benefits are a key requirement. A current example is the foreseen implementation of SDMX in the context of FRIBS.
• To be truly effective, a more comprehensive strategy for content harmonisation that goes beyond data collection and exchange is needed.
![Page 22: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/22.jpg)
Next steps
![Page 23: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/23.jpg)
Next steps
• Q4/2018: Eurostat will draft a plan for introduction of the light implementation package in all the domains that have not yet implemented SDMX.
• Item 3.3 of the agenda of the Standards Working Group discusses some proposals and ideas to move forward on the management of SDMX code lists.
![Page 24: SDMX implementation: Status and next steps Item 3.2 ... · Luca Gramaglia, Unit B5 . SDMX and its benefits . Technical standards (Information Model and formats) Statistical guidelines](https://reader031.vdocuments.us/reader031/viewer/2022022801/5c6f300309d3f29e208bf173/html5/thumbnails/24.jpg)
Thank you for your attention!
Any questions