Big Data, can you tell me where I want to go from how? to delivering

ESS Modernisation Workshop Bucharest, 16-17 Mar 2016

Eurostat

Big data at Eurostat

ESS (European Statistical System) Scheveningen Memorandum Sep 2013

Task Force Big Data Big Data Roadmap and Action Plan June 2014

May 2015 Big Data Business Cases May 2015 BIGD Project Nov 2015 Big Data ESSnet Jan 2016 Contract on supporting services

ESS Vision 2020

European Commission Communication Data4Policy Networking Group

International cooperation (UNSD, UNECE, etc.) • UN/ECE project “Big data in official statistics” (Sandbox) • UNSD Global WG on Big Data

2

Scheveningen

Eurostat

Key areas

Modernisation: ESS Vision •exploiting the potential on new data sources •establishing alliances and partnership with data owners •new IT tools and methodological development •organisational changes •improving existing data collection methods

Eurostat

Eight enabling projects + supporting frameworks

Supporting Frameworks: • Enterprise Architecture • Quality • Cooperation Models

Eurostat

What is the BIGD project? • What?

• Implement ESS Big Data Action Plan • Why?

• Enable the ESS to gradually integrate big data sources into the production of European and national statistics

• How? • Concrete Big Data Pilots • In parallel work on horizontal issues

• Three time ranges in roadmap • Short term 2015-2016

Analysis of legislation, Strategy, Ethics, Communication • Medium term 2016-2020

Pilots, Partnerships, IT Architecture, Skills • Long term >2020

Full integration into official statistics (?!)

Eurostat

… and the ESSnet Big Data?

• Supports ESS Big Data Action Plan / BIGD project Preparation starts 2015 based on FPA/SGA construction

• Core of work: 3+3 pilots 2016-2017 / 2017-2018 “learning by doing”

• Exploration of specific data sources e.g. mobile communication data, social media, internet sites, ….

• Issue of multisource Statistics • Contributes to generic frameworks and guidelines

e.g. methodology, quality, legislation, IT, communication, architecture, …

Eurostat

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics / Communication

Partnerships

Pilots

Action Plan Themes

Eurostat

Experience sharing

Big Data project Approach

Policy Legislation

Quality Skills Methodology IT Infrastructure

Ethics / Communication

P

I

L

O

T

S

Eurostat

Actions

Official Statistics Big Data Strategy Roadmap and Action Plan

European Commission Communication "Towards a thriving data driven economy"

Private Public Partnership on big data

Data4Policy Initiative

Data Revolution at UN level

Challenges ▫ Datafication ▫ Impact on Official Statistics ▫ Answer of Statistical System ▫ Reaction of Government

10

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics /

Communication

Partnerships

Pilots

Eurostat

Actions Big Data ESSnet: Pilot projects

2015 – 2019 (FPA/SGA construction) Exploring different big data sources

Characteristics of data, potential outputs, Methodology and Quality, IT requirements

Partnerships with data providers, academia, users Cooperation with UN

Big Data Competition Big Data Workshop in Oct 2016

Challenges ▫ Exploration & tentative implementation ▫ Scientific Approach ▫ Cooperation, sharing of know-how ▫ Access to data, lack of skills, new user

demands

11

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics /

Communication

Partnerships

Pilots

Pilots Communication

Mobile Communication

Social Media

WWW

Web Searches

Analysis of websites' contents • Job

Advertisements • Businesses'

Websites

• E-Commerce • Real estate

Internet Traffic

Sensors

Traffic loops

Smart meters

Automatic Vessel

Identification System

Satellite Images

Webcams

Process generated data

Reservation Systems • Flight Booking

transactions

•Trains •Hotels

Supermarket Cashier Data

Loyalty Cards

Financial transactions

Mobile Payments

eGovernment

Crowd sourcing

Voluntary Geographic Information

websites (OpenStreetMap)

Voluntary Information

websites (Wikipedia)

Community pictures collection

Eurostat

▫ List of pilot projects (Specific Grant Agreement) Web scraping

job vacancies ; enterprise characteristics

Smart meters electricity consumption ; temporary vacant dwellings

Automatic Identification System (Ships) vessel identification data

Mobile phone data Preparing for Access to data

Scenario for using multiple inputs

13

ESS Big Data Pilots

Eurostat

Actions ▫ Cooperation with UN on quality and methodological framework

for big data ▫ Transversal topics in ESSnet pilots projects ▫ Generalisation to frameworks

Challenges ▫ Transversal challenges to all big data

activities: quality, methodology ▫ Multiple sources for multiple outputs ▫ Sound methodology ("from design-

based to model-based approach") ▫ Big data vs. statistics : "goodness of

fit" (concepts, representativeness,…)

14

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics /

Communication

Partnerships

Pilots

Eurostat

Actions ▫ Training Strategy

▫ Competency based approach ▫ Program for European statisticians (ESTP)

▫ In the next years: dedicated courses on big data ▫ Focus on big data sources and on big data tools ▫ Acquiring the skills needed to assess sources and their

quality, the skills to use tools and to explore big data sources

Challenges ▫ New skills for staff: statisticians

vs. data scientists ? ▫ Relations with external data

providers ▫ Fast changing conditions and

situations

15

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics /

Communication

Partnerships

Pilots

Eurostat

ESTP courses supporting big data (2016)

16

Introduction to big data and its

tools

Hands-on immersion on big

data tools

Big data sources - Web, Social media and text analytics

Advanced big data sources - Mobile phone and other

sensors

Big data courses

Can a statistician become a data

scientist?

The use of R in official statistics:

model based estimates

Time-series econometrics

Methodology courses

Nowcasting

Activity

29 Feb – 2 Mar 21 – 24 Jun

7 – 10 Nov

12 – 15 Sep

5 – 7 Apr 8 – 10 Jun 24 – 26 Feb

Eurostat

Actions ▫ Contract

▫ 2016-2017 (22 months) ▫ Selected big data sources

▫ Analysis of legislative situation in EU ▫ Review of ethical principles, ▫ Elaboration of Communication Strategy ▫ See also the Feasibility study on the use of mobile positioning data for tourism

statistics (report on feasibility of access)

Challenges ▫ Access to data & continuity of

access ▫ Data security & privacy concerns ▫ Use of data only for statistical

purposes ▫ Impact on the public opinion of

privacy and security concerns ?

17

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics /

Communication

Partnerships

Pilots

Eurostat

Actions ▫ Sandbox IT infrastructure for experimenting

UNECE, Eurostat, EU

New Software (Hadoop, Pig, Python, …) Pilots for gaining experience

Challenges ▫ Computing capacity, hardware ? ▫ Analytical tools, software? ▫ Storage ?

18

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics /

Communication

Partnerships

Pilots

Eurostat

Thank you for your Attention!

To be honest, Mr Wirthmann, I don't care how many yottabytes you have