inspire australian initiatives for standardising statistical processes and metadata simon wall...

InSPIRe

Australian initiatives for standardising statistical processes and metadata

Simon Wall

Australian Bureau of Statistics

December 2012

1

2

Increasing cost & difficulty of

acquiring data

New competitors &

changing expectations

Rapid changes in the environment

Competition for skilled resources

Reducing budget

Riding the big data wave

Challenges facing NSIs

• Reduce the cost and time of doing business• Grow the business through new statistical products

and services• Deliver the first large scale digital Census (2016) on

time, to budget and quality

while delivering Business As Usual.

3

Challenges facing ABS

• Through large scale Innovation across the whole ABS

• We will:

– radically transform the way we acquire, collate, use, reuse and disseminate statistical information

• By:

– industrialising, modernising and reengineering our business processes

– reengineering our statistical infrastructure and the way we manage information

– develop capability needed to meet future needs

– Collaborating with other international NSIs4

How will we get there?

InSPIRe

5

What is InSPIRe?

• Infrastructure for Statistical Process and Information Management Re-engineering

• Building core infrastructure in the ABS for:– Business Process management

– Information management

• SWM – Statistical Workflow Management System

• MRR– Metadata Registry and Repository

6

SWM

• The Statistical Workflow Management System

• An environment that will be used to develop and manage common business processes – These processes can drive and/or be used by

tools and applications.

7

MRR

• The Metadata Registry and Repository consists of two parts:

– Repository • The centralised ‘bucket’ to store standards

based metadata.

– Registry• The catalogue that lets you find out what is in

the repository.

8

MRR

• Registers and stores a wide range of information, including:– Metadata (i.e. Classifications, Variables, etc)

– Data – registered in MRR, stored in EDW

– Process Definitions and Paradata – information to run a process

– Process Metrics – information about a process that was run.

9

MRR

• The MRR– Enforces registration of the information, ensuring consistent

documentation– Reads information in a range of standard supported metadata

formats• (initially DDI and SDMX)

– Re-issues metadata in whatever standard format is required by a given process.

• Regardless of the format in which the metadata was originally created in.

– This information is stored for use and later re-use by the metadata driven processes in SWM

– Enables the searching and discovery of metadata for re-use.

10

Interfaces

• Processes and Systems will interact with the MRR through a web service interface that supports a controlled set of standards.– Currently limited to DDI 3.1 and SDMX 2.1

11

InSPIRe

SWMSWMSearchSearch

Business ProcessBusiness Process

Users

User InterfaceUser Interface

MRRMRR

RetrieveRetrieve RegisterRegister

Benefits of InSPIRe

• Basis for greater automation and reuse of processes and metadata

• Faster to market– For individual collections– For building new collections

• Cheaper to market (long term)– For individual collections– For building new collections

13

Progress so far…

• Proof of Concept Project (2010/2011)– A group of simple use cases to prove the idea of the

MRR and SWMs.

• Demonstrating:– Benefits

– Potential functionality

– What is technically achievable

• Building capability

14

Pathfinders

• Pathfinder Integration (July 2012)– Expanded on PoC

– ‘Pathfinder’ projects to produce further use cases for the MRR and SWMs.

• Not end-to-end processes, but isolated ‘snap shot’ projects to demonstrate the capabilities of InSPIRe.

• Four pathfinders chosen– Including web data capture (e-forms) and REEM (Remote

Execution Environment for Microdata)

REEM and Web Data Capture Pathfinders

• Proved integration with real ABS systems

• Demonstrated the benefits of integration with InSPIRe, namely:

– Re-use of processes (in SWM)

– Standards based metadata (DDI in MRR) driving business tools

– Automating business processes (in SWM)

Early Adopter Projects

• Projects which are positioning to integrate with InSPIRe within the next 12 months.– Includes projects focusing on administrative data,

e-forms and the 2016 Census.

• Suitable candidates to prove the operation InSPIRe.

– implementing processes in different phases of the GSBPM.

17

What we have achieved so far

• MRR– Design and review, mappers, shredders, database,

automated generation of metadata types, basic search, registry model

• SWM– Environment setup, test cases with projects

• Capability Development– Best practice guidelines for implementing DDI, metadata

content guidelines, InSPIRe integration information packs

18

What we still have to do

• MRR– Production version

• SWM– Further process definition, integration with other projects

• Metadata Authoring Environment– A mechanism for:

• Taking existing metadata content and registering it in the MRR

• Creating new metadata content via a content creation tool and registering it in the MRR

19

The future…An example

20

21

Survey Designer

Retrieve Store

Content Creation

Tool

e-form Transformation

EnvironmentWeb form

MRR

DDI DDI

Register

Data Collection

DDI

EDW

Retrieve StoreSWM