1
SDMX Reference Infrastructure(SDMX-RI)
Work in progress, status and plans
Bengt-Åke Lindblad, Adam Wroński
Eurostat
Eurostat Unit B3 – IT and standards for data and metadata exchange
Content
What is SDMX-RI Why use SDMX-RI Usage status Development strategy Release calendar
Eurostat Unit B3 – IT and standards for data and metadata exchange2
What is SDMX-RI
Universal framework for modern data provision Set of pick-and-choose reusable building blocks
allowing a statistical office to expose data to the external world based on access rights
Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse
Uses SDMX standards incl. one for Web Services
Eurostat Unit B3 – IT and standards for data and metadata exchange3
Eurostat Unit B3 – IT and standards for data and metadata exchange4
Why SDMX-RI
Developed to simplify the exchange of data Provides standard software and components,
allowing individual statistical organisations to interact and exchange their data using the same software and methodology
Modular approach, use part or the entire infrastructure, extend it by adding new modules or modify it in any other way to suit their own purposes
Developed in both Java and .NET
Eurostat Unit B3 – IT and standards for data and metadata exchange5
Usage
As of Sept 2012 deployed in 22 EU countries Tested across EU Member States (2011
population census) Running in Mexico Expression of interest
Latin America, the Caribbean, OECD and Russia July 2013:
expected to run in EU27 member states in Eurostat dissemination Web Service upgrade
Eurostat Unit B3 – IT and standards for data and metadata exchange6
SDMX-RI versions
StreamingSDMX-RI
Initial Solution
Common API SDMX
v2.1
SDMX-RIIntermediate
Solution
SDMX-RIUltimate Solution
Eurostat Unit B3 – IT and standards for data and metadata exchange7
Development strategy 2012-2013
Architectural changes are needed (two step approach) Intermediate solution to solve performance & out of
memory errors Ultimate solution to provide a common API and
implement SDMX 2.1 Implement new user requests and correct defects Widen the scope and usage of SDMX-RI among data
providers ESSVIP project such as Web INfrastructure for Statistics Reuse for other statistical data collections DSWS (Eurostat dissemination web service)
Eurostat Unit B3 – IT and standards for data and metadata exchange8
Why an intermediate solution
Problems needed to be solved within short time for existing users Performance
High memory consumption and response times constrained organizations from putting it in production
Out Of Memory errors Impossible to serve large data requests.
Eurostat Unit B3 – IT and standards for data and metadata exchange9
What changed?
Streaming of data in the service Migration to JAX-WS in Java
IO revised with Streaming Writers Data Retriever API changed (streaming)
Eurostat Unit B3 – IT and standards for data and metadata exchange10
Benefits
Better performance Improvement up to 4 times faster for .Net Improvement up to 7 times faster for Java
Solution to Out Of Memory problems for large datasets No memory constrains Data are streamed to the client
Eurostat Unit B3 – IT and standards for data and metadata exchange11
Impact
Organizations that have installed initial solution Only re-install the Web Service Package Existing users of Web Service package not affected
Same v2.0 interface remain Organizations using the components API
API has changed due to streaming support Migration required
Organizations has done modifications to the source code Have to make the changes again if they want to use the
intermediate version.
Eurostat Unit B3 – IT and standards for data and metadata exchange12
Why an ultimate solution?
Use of common API (that supports various versions and flavors of SDMX) Interfaces which allow implementation using the
components and also the interfaces for component-to-component communication
Better inter-organization component reusability Ability to support new versions of SDMX without
changing our applications Support of SDMX 2.1
New messages (data representation, queries) New Interfaces (SOAP/REST)
Eurostat Unit B3 – IT and standards for data and metadata exchange13
What will be changed?
All modules migrated to the common API To use of new SDMX Model/IO Components APIs will be changed
Support of SDMX 2.1 messages and new query features Extend to support 2.1 standardized SOAP and RESTful APIs
New Web Service end points Co-existence with v2.0 service Support of v2.1 error codes
Eurostat Unit B3 – IT and standards for data and metadata exchange14
Impact
Mapping Assistant users will be able to reuse their Mapping Stores Mapping Store upgrade will be supported Will be required to re-install MA only
Organizations that have installed initial solution Only to re-install the Web Service Package Existing users of Web Service package will not be affected
Same v2.0 interface will remain Organizations using components API
API has changed due to streaming support Migration will be required Migration guidelines will be provided
Organizations done modifications to the source code Will have to make the changes again if they want to use the
new version.
Eurostat Unit B3 – IT and standards for data and metadata exchange15
Status - release Common API development (jar/dll expected end 2012)
Java: ongoing .NET: not started yet (API should be ported to C# and then
implemented) SDMX 2.1 support (expected Q1 2013)
Not yet started (dependent on API) Enhancements and bug corrections (independent)
Mapping Assistant (Sept 2012) Improved transcoding, error handling, QueryEditor, etc
Mapping Assistant (end 2012) Implementation of 11 request mainly linked to the user interface
and the usage Test Client
using encrypted user id and password
Eurostat Unit B3 – IT and standards for data and metadata exchange16
SDMX-RI contacts
Eurostat Unit B3 – IT and standards for data and metadata exchange17
Web Service Package extension to SDMX 2.1 and new interfaces
ControllerController
Data Retriever (streaming)
Data Retriever (streaming)
Web
S
ervi
ce
Pro
vide
r
Common API
Common API
Structure RetrieverStructure Retriever
NSI_Service_2.0NSI_Service_2.0 NSI_Service_2.1NSI_Service_2.1
SOAP Request v2.0
SOAP Request v2.1
NsiRestServiceNsiRestService
REST Request v2.1
API Implementation
API Implementation
Eurostat Unit B3 – IT and standards for data and metadata exchange18