sdmx overview nsf accounting interoperability workshop may 12-13 washington dc arofan gregory...

SDMX Overview

NSF Accounting Interoperability WorkshopMay 12-13

Washington DCArofan Gregory (arofan.gregory@earthlink.net)

Rene Piche (rpiche@imf.org)

What is SDMX?

• The problem space:– Statistical collection, processing, and

exchange is time-consuming and resource-intensive

– Focus on aggregate data (esp. time series)– Various international and national

organisations have individual approaches for their constituencies

– Uncertainties about how to proceed with new technologies (XML, web services, SOA …)

What is SDMX?

The Statistical Data and Metadata Exchange (SDMX) initiative is taking steps to address these challenges and opportunities that have just been mentioned:– By focusing on business practices in the field

of statistical information– By identifying more efficient processes for

exchange and sharing of data and metadata using modern technology and open standards

International OrganisationsRegional Organisations

accountsstatistics

Banks, CorporatesIndividual Households

trans-actions

accounts

National Statistical Organisations

accountsstatistics

180

+ C

ount

ries

180

+ C

ount

ries

Inte

rnet

, S

earc

h, N

avig

atio

nIn

tern

et,

Sea

rch,

Nav

igat

ionwww.z.org

www.hub.org

www.x.org

www.y.org

Historical Note

• SDMX uses an approach based on the 10-year-long success of an earlier standard – GESMES/TS

• GESMES/TS was an initiative that is used today in many countries for collecting, exchanging, and updating statistical databases– GESMES/TS is now SDMX-EDI

• Focus is on time-series, and is mostly used by central banks

Who is SDMX?• SDMX is an initiative made up of seven

international organizations:– Bank for International Settlements– European Central Bank– Eurostat – International Monetary Fund– Organisation for Economic Cooperation and

Development– United Nations– World Bank

• The initiative was launched in 2002

SDMX Products• Technical standards for the formatting and

exchange of aggregate statistics:– SDMX Technical Specifications version 1.0

(now ISO/TS 17369 SDMX – TC 154 WG2)– SDMX Technical Specifications version 2.0

(soon to be submitted to ISO – TC 154 WG2)

• Content-Oriented Guidelines (in draft)– Common Metadata Vocabulary– Cross-Domain Statistical Concepts– Statistical Subject-Matter Domains

Detailed SDMX Goals• Reduce national reporting burden to international institutions• Fostering consistency, accuracy, and timeliness of data and

metadata dissseminated by national and international institutions, relying on what is decentrally released via national websites

• Enhancing national statistical processing efficiency, especially through internationally-recognised standard formats for exchanges between statistical silos within institutions and with other national statistical agencies

• Providing standards for web-based dissemination formats that are computer readable and facilitate updating of databases

• Enhancing comparison of data and metadata analysis through standard formats and content-oriented guidelines

Recent Events

• Jan 2007 – Launch meeting at the World Bank for SDMX 2.0 Technical Specifications

• February 2007 – Endorsement of SDMX by EU’s Statistical Programme Committee

• March 2008 – SDMX becomes the preferred standard for data and metadata of the UN Statistical Commission

Adoption

• By most estimates, the SDMX user base doubled in 2007

• Expectation is that growth will accelerate in 2008

• Areas of adoption include financial and monetary statistics, but also include many other domains: education, labor, population, health, etc.

Adopters/Interest• The following are known adopters (or planning to adopt):

– US Federal Reserve Board and Bank of New York– European Central Bank– Joint External Debt Hub (WB, IMF, OECD, BIS)– UN/TRADECOM at UN Statistical Division– NAAWE (National Accounts from OECD/Eurostat)– SODI (Eurostat and European Governments)– Mexican Federal System– Vietnamese Ministry of Planning and Investment– Qatar Information Exchange– IMF (BOP, SNA, SDDS/GDDS)– Food and Agriculture Organization– Millenium Development Goals (UN System, others)– International Labor Organization– Bank for International Settlements (FSI, many others)– OECD– World Bank– Marchioness Islands (Spanish/Portugese Statistical Region)– UNESCO (Education)– Australian Bureau of Statistics– Statistics Canada– There are probably others we are not aware of

Scope

• SDMX covers three major areas:– (Meta)model and XML formats for aggregate

data– (Meta)model and XML formats for “reference”

and “provisioning” metadata– Registry-based, distributed architecture

• Technical specifications and “statistical” aspects are cleanly separated

The SDMX Information Model

• SDMX is a model-driven standard– All technology implementations are an

implementation of a common meta-model– All concepts within SDMX for data and

metadata are configurable– SDMX works with any type of classification or

“ontology”– Instances of the metamodel are expressed in

a dedicated XML – SDMX-ML

SDMX-ML

• SDMX-ML is the set of XML interfaces/messages which provide:– Processable metadata (for data structures, for

metadata structures, for data and metadata provisioning)

– Data formats– Metadata formats– Registry interactions (metadata queries,

registration, classifications, concepts, etc.)– Data and metadata queries

Alignment with Other Standards

• SDMX was designed to fit into the standards environment:– Web-services and SOA standards (WSDL,

SOAP, WSS, etc.)– ISO/IEC 11179 & ebXML CCTS– ebXML Registry/Repository specifications– Data Documentation Initiative (DDI)

• SDMX and XBRL are working to produce a standard cross-walk

Support for Other Standards

• The SDMX distributed architecture is designed to work with related standards– Generic ability to model other (standard)

processes– Generic ability to register other (standard)

metadata and data formats– Ability to integrate with an ISO/IEC 11179

semantic registry

• Integration with sources and outputs is a design feature of SDMX

High-Level Vision – Standards MappingsFederated Registries (Based on SDMX, ebXML, web services)

Aggregated Data/Metadata

(SDMX)

XBRLBusiness Reports

DDI Microdata

Sets

ISO 19115Geographies

Dublin CoreCitations

Used in

registered

References to source data

Standard classifications

Organizedusing

ISO 11179

Semantic definitions

METS Packaging

Standards Alignment

• To support the submission of the SDMX Technical Specifications to ISO, papers will be provided covering several mappings/alignments:– XBRL– ISO/IEC 11179– ebXML CCTS– ebXML Registry/Repository– Data Documentation Initiative (DDI)

Questions?