Dealing with Open Data at ISTAT

Giovanni A. BarbieriStatistics Italy (Istat)

The Open Data Movement

Our proposal is simple: […] the federal government […] is to provide data that is easy for others to reuse, rather than to help citizens use the data in one particular way or another

Open infrastructures that enable citizens to make their own uses of the data

Reverse the current policy, which is to regard government websites themselves as the primary vehicle for the distribution of public data, and open infrastructures for sharing the data as a laudable but secondary objective [Robinson,Yu, Zeller and Felten 2008]

2

Crowdsourcing Government Transparency

Government information that is nominally publicly available is in fact difficult to access either because it is not online or, if it is online, because it is not available in useful and flexible formats [Brito 2008]

“Structured data”

Associated structured XML file would allow a user to sort the data by ascending or descending date, alphabetically by headline or author, by number of words, and in many other ways

3

Open Data Ecosystem

“Open data are adding a new dimension to big data analytics and

giving rise to novel, data-driven innovations.”

(McKinsey Global Institute Report, Oct. 2013)

From Citizens IBM BLOG

Wide Range of Open Data Consumers

Citizens that would like to learn characteristics of the places they are, e.g.

with mobile apps showing location-specific features

Journalists that need to access data for updated and aware information

communication

Educators that are helped in theirteaching task by access to data on

different application domains

4

Official Statistics meets Open Data

Official Statistics can more easily reach such wide

range of users if conveyed through open data

Recent technologies advances in the open data

community enable new advanced dissemination

channels for Official Statistics

5

Reinforcingtrust

Getting closerto users

Reaching new users

Givinginformation

back

Improvingmetadata

Linked Open Data

Semantic Web Technological

Standards

OWL

Knowledge Representation

Linked Open Data - LOD

6

Why is Linked Data an Opportunity?

Linked Data as a semantically rich paradigm for data representation

Rich enough for the strict requirements of Official Statistics

Formal and well-defined data structures, i.e. ontologies

Linked Data as an international standard (W3C) Tools availability and independence Beyond statistical users RDF: Resource Description Framework (W3C)

(subject-predicate-object)

7

Istat’s Linked Open Data Portal - 1

Istat LOD Portal: http://datiopen.istat.itEnglish Version: http://datiopen.istat.it/index.php?language=eng

8

Platform for• Selecting• Navigating• Searching• Querying• VisualizingOpen Data

The platform allows• Direct access to data via Web Services • M2M solutions (e.g. GIS-LOD) • Data conversion• Export to productivity tools• Visualization by means of external tools

Istat’s Linked Open Data Portal - 2

9

STEP 1 STEP 3 STEP 2

Istat’s Linked Open Data Portal - 3

Steps to a «perfect» data portal

10

Istat’s Linked Open Data Portal - 4

Guided Access

Freedom of access

Typ

eo

f in

tera

ctio

n

Free AccessHu

man

b

asic

Mac

hin

e To

M

ach

ine

Navigation

Guided queries

Query REST onSPARQL EndPoint

Query via SPARQL EndPoint

Web Service

Download

Hu

man

A

dva

nce

d

11

Predefined Queries

(Set of simple and customizable queries)

Free Queries

(SPARQL Queries)

Navigation

Guided Queries

Download

Typ

eo

f in

tera

ctio

n

Hu

man

Guided Access Free Access

Interaction Modes

Freedom of access

Basic Advanced(Human) Usertechnical skills Intermediate

Free Query via SPARQL EndPoint

12

Use Case 1: Spatial Querying

App that displays on a map some population indicators of the

nearest census sections to specific GPS coordinates

LOD when accompanied by spatial information allow to access data

using spatial queries

13

Use Case 2: Federated Querying - 1

Federated query on Istat and ISPRA, i.e. the query accesses Istat and ISPRA portals

With LOD, it is very easy to compare data coming from different sources

(linked for example at territorial level)

Query on onePortal

Results dynamicallyretrieved

from both portals

14

ISPRA - The Italian National

Institute for Environmental

Protection and Research

Istat

Use Case 2: Federated Querying - 2

ISTAT data:

Census BuildingsISPRA data:

Data on land use / soil consumption

Example Query:

Municipality-level analysis of land use / soil consumption and number of

buildings by period of construction

Dynamically generated!

15

Use Case 3: Istat as Open Data Provider in SPOD

Social discussion (on the left) about a graphic representationof Census Data (on the right)

Dynamicallygenerated!

16

SPOD: Social Platform for Open Data

Conclusions

A dissemination strategy based on open data does

put the Official Statistics users at the centre:

Reaching them through different channels

e.g. apps and social media

Making easier for them to retrieve data

e.g. federated queries that make transparent

the distribution of data on different portals

Providing richer services to them

e.g. spatial querying and dynamical

visualizations

17

Open data and in particular Linked Open Data have a leading role in data innovation

for Official Statistics

• Macroscale vs microscale modeling

– Pseudo-Einstein (as simple as possible but not simpler)

– Von Neumann (agent-based modeling)

• Technological constraint enabling technology

• It widens the space of what is feasible:

– In production: our experience with SBS.Frame

– In analysis and research…

• A paradigm shift?

– Statistical mechanics vs agent-based modeling

– Just because you can doesn’t mean you should

• Back to open data

– From dissemination to release (“data liberation” at StatCan) to the

development of information

– The regulators need to introduce new rules in line with the new

scenarios (Don't think of an elephant!: know your values and frame the

debate)

One More Thing

18

Thanks to all my colleagues in Istat contributing to

the LOD Portal

Special thanks to Monica Scannapieco and

Stefano De Francisci

Questions and clarifications: contact me at

barbieri@istat.it

Acknowledgments and Thanks

18