Implementation of GSBPM, DDI and SDMX reference metadata at Statistics Denmark

UNECE workshop on International Cooperation for Standard-based Systems

Geneva 5-7 May 2015

Mogens Grosen Nielsen Statistics Denmark(mgn@dst.dk)

Agenda

1. History, strategy and principles on quality and metadata

2. Definitions, models and solution on quality reporting

3. Towards more integrated metadata4. Information models 5. Lessons learned and opportunities

Short history on metadata initiatives • Before 2010: Separated and partially standardised

metadata systems• January 2010: Taskforce: integration variables,

classifications, statistical concepts and quality• October 2011: Test DDI (focus on integration)• January 2012: EU grant. SDMX quality concepts,

DDI, and GSBPM. Colectica as tool• January 2015: Quality declarations for all statistics

(300+) in Colectica• February 2015: Strategy on quality and metadata

approved

Strategy: Vision

1. Metadata about content and quality must guide internal and external users

a) in their knowledge processes b) give precise information about the

products

2. Internal efficiency gains3. International standards

GSBPM, SDMX, GSIM, DDI and others

4

5

Business Process Management (End-to-End Processes)

Stepwise implementation Code of Practice and Quality Assurance Framework Principles on metadata

Metadata must fulfill user needs Metadata and metadata flow integrated into GSBPM As much reuse as possible Active use of metadata in IT-systems (incl. metadata driven

production)

Strategy: Principles

Metadata definition #1. How to communicate this term to statisticians?

Use the SDMX definition as the short and easy-to-understand definitionReference metadata: • Conceptual metadata • Methodological and processing metadata • Quality metadataStructural metadata: • Metadata act as identifiers and

descriptors of the data

Metadata definition #2. How to communicate this term to statisticians? Use Generic Statistical Information Model (GSIM) to

tell the full story

”Classical” metadata using Data Documenation initiative (DDI), SDMX and Colectica – ”The Diamond”

Hvad betyder

Quality declaration

Variable/datasetConcept

Variable database

Klassifikationsdatabase

Classifications

Methods/ ”Survey”StatBank Methods papers

Class database

Concepts database

Implemented in 2012-2015

9

SDMX Standard for Quality Single Integrated Metadata Structure (SIMS)

Content (population, concepts, reference time etc) Statistical processing (sources and methods) Information on 5 quality dimensions

1. Relevance

2. Accuracy and reliability

3. Timeliness and punctuality

4. Comparability

5. Accessibility and Clarity

SIMS and reporting formats Euro-SDMX Metadata Structure (ESMS) and ESS Standard for Quality Reports Structure (ESQRS)

10

11

GSBPM and work processes with focus on quality declarations

Needs Prepare user needs etc.

Analyse : Fill in accuracy

etc.

METADATA IN

COLECTICA

Enter Quality Information Publish at Dst.dk

Quality eports to Eurostat

Publish at the Intranet

Existing metadata

The solution

METADATA IN

COLECTICA

Enter Quality Information Publish at Dst.dk

Quality reports to Eurostat

Publish at the Intranet

Existing metadata

300 surveys implemented January 10 2015

• Business perspective: Business Process Management (BPM) and metadata-driven approach

• GSIM compliant model in DDI and Colectica (concepts, variables, classification etc)

• Harmonisation of statistical concepts• Integration of metadata in publishing systems • Metadata management

Towards more integration of metadata

Users

User needs /orders

General Environment: Political/legal context, Technology/standards

Business Process Management

?

Respondents/ registers etc

Ressources:staff IT-systems etc

General Environment context: Political/legal, Technology/standards

Respondents/ registers etc

Ressources:staff IT-systems etc

Management-, core- and support-processes

Management processes

Support processes: Quality, metadata, methods & IT

Users

User needs /orders

17

Business processes and metadata driven approach

18

Business processes and metadata driven approach

Reference metadata recorded: needs, purpose etc.

19

Business processes and metadata driven approach

• Structural metadata: DDI on questionnaire, variables and cubes etc

• Reference metadata: concepts, population etc

20

Business processes and metadata driven approach

Metadata used to create survey system, databases and output systems etc

21

Business processes and metadata driven approach

Auto-generated survey system used

22

Reference metadata on quality etc

Business processes and metadata driven approach

23

• Structural and reference metadata used for dissemination (e.g. quality reporting)j

• Autogenerated code used in dissemination systems

Business processes and metadata driven approach

24

Business processes and metadata driven approach

All metadata used for evaluation

What we are doing at Stat DK

Models: conceptual, logical and physical

Model / level What are we doing at Stat DK

Conceptual Selection of variable, concept etc from GSIM (the concept corner)

Logical GSIM compliant DDI model (3.2)

Physical GSIM compliant DDI model implemented in Colectica

Need for • Improving content - declarations more uniform and

compliant with common guidelines• Using the same quality-concepts for many purposes:

report to Eurostat, publish at national web-site and to other international organisations.

• Need for more analysis and improved dissemination at www.dst.dk

• Improved focus on change management and communication

• Clear organisational roles and a dedicated cross-cutting group to introduce and implement standards

Lessons learned

- Work on models – from abstract to concrete level (BPM, GSIM, GSBPM, DDI and SDMX)

- GAMSO and CSPA needs more attention- Sharing of solutions for input, processing and

output systems – e.g. Colectica add-ins- How to handle metadata management

integrated into GSBPM- Common metadata

Opportunities for cooperation

Thanks for your attentionRemember:• DDI conference in Copenhagen 2-3 December 2015

The End!