Big Data, can you tell me where I want to go from how? to delivering

ESS Modernisation Workshop Bucharest, 16-17 Mar 2016

Eurostat

Big data at Eurostat

ESS (European Statistical System) Scheveningen Memorandum Sep 2013

Task Force Big Data Big Data Roadmap and Action Plan June 2014

May 2015 Big Data Business Cases May 2015 BIGD Project Nov 2015 Big Data ESSnet Jan 2016 Contract on supporting services

ESS Vision 2020

European Commission Communication Data4Policy Networking Group

International cooperation (UNSD, UNECE, etc.) UN/ECE project Big data in official statistics (Sandbox) UNSD Global WG on Big Data

2

Scheveningen

http://epp.eurostat.ec.europa.eu/portal/page/portal/pgp_ess/0_DOCS/estat/SCHEVENINGEN_MEMORANDUM Final version_0.pdfhttp://www.cros-portal.eu/content/ess-big-data-action-plan-and-roadmap-10http://www1.unece.org/stat/platform/display/bigdata/Big+Data+Project

Eurostat

Key areas

Modernisation: ESS Vision exploiting the potential on new data sources establishing alliances and partnership with data owners new IT tools and methodological development organisational changes improving existing data collection methods

Eurostat

Eight enabling projects + supporting frameworks

Supporting Frameworks: Enterprise Architecture Quality Cooperation Models

Eurostat

What is the BIGD project? What?

Implement ESS Big Data Action Plan Why?

Enable the ESS to gradually integrate big data sources into the production of European and national statistics

How? Concrete Big Data Pilots In parallel work on horizontal issues

Three time ranges in roadmap Short term 2015-2016

Analysis of legislation, Strategy, Ethics, Communication Medium term 2016-2020

Pilots, Partnerships, IT Architecture, Skills Long term >2020

Full integration into official statistics (?!)

Eurostat

and the ESSnet Big Data?

Supports ESS Big Data Action Plan / BIGD project Preparation starts 2015 based on FPA/SGA construction

Core of work: 3+3 pilots 2016-2017 / 2017-2018 learning by doing

Exploration of specific data sources e.g. mobile communication data, social media, internet sites, .

Issue of multisource Statistics Contributes to generic frameworks and guidelines

e.g. methodology, quality, legislation, IT, communication, architecture,

Eurostat

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics / Communication

Partnerships

Pilots

Action Plan Themes

Eurostat

Experience sharing

Big Data project Approach

Policy Legislation

Quality Skills Methodology IT Infrastructure

Ethics / Communication

P

I

L

O

T

S

Eurostat

Actions Official Statistics Big Data Strategy

Roadmap and Action Plan

European Commission Communication "Towards a thriving data driven economy"

Private Public Partnership on big data

Data4Policy Initiative

Data Revolution at UN level

Challenges Datafication Impact on Official Statistics Answer of Statistical System Reaction of Government

10

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics /

Communication

Partnerships

Pilots

Eurostat

Actions Big Data ESSnet: Pilot projects

2015 2019 (FPA/SGA construction) Exploring different big data sources

Characteristics of data, potential outputs, Methodology and Quality, IT requirements

Partnerships with data providers, academia, users Cooperation with UN

Big Data Competition Big Data Workshop in Oct 2016

Challenges Exploration & tentative implementation Scientific Approach Cooperation, sharing of know-how Access to data, lack of skills, new user

demands

11

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics /

Communication

Partnerships

Pilots

Pilots Communication

Mobile Communication

Social Media

WWW

Web Searches

Analysis of websites' contents Job

Advertisements Businesses'

Websites

E-Commerce Real estate

Internet Traffic

Sensors

Traffic loops

Smart meters

Automatic Vessel

Identification System

Satellite Images

Webcams

Process generated data

Reservation Systems Flight Booking

transactions

Trains Hotels

Supermarket Cashier Data

Loyalty Cards

Financial transactions

Mobile Payments

eGovernment

Crowd sourcing

Voluntary Geographic Information

websites (OpenStreetMap)

Voluntary Information

websites (Wikipedia)

Community pictures collection

Eurostat

List of pilot projects (Specific Grant Agreement) Web scraping

job vacancies ; enterprise characteristics

Smart meters electricity consumption ; temporary vacant dwellings

Automatic Identification System (Ships) vessel identification data

Mobile phone data Preparing for Access to data

Scenario for using multiple inputs

13

ESS Big Data Pilots

Eurostat

Actions Cooperation with UN on quality and methodological framework

for big data Transversal topics in ESSnet pilots projects Generalisation to frameworks

Challenges Transversal challenges to all big data

activities: quality, methodology Multiple sources for multiple outputs Sound methodology ("from design-

based to model-based approach") Big data vs. statistics : "goodness of

fit" (concepts, representativeness,)

14

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics /

Communication

Partnerships

Pilots

Eurostat

Actions Training Strategy

Competency based approach Program for European statisticians (ESTP)

In the next years: dedicated courses on big data Focus on big data sources and on big data tools Acquiring the skills needed to assess sources and their

quality, the skills to use tools and to explore big data sources

Challenges New skills for staff: statisticians

vs. data scientists ? Relations with external data

providers Fast changing conditions and

situations

15

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics /

Communication

Partnerships

Pilots

Eurostat

ESTP courses supporting big data (2016)

16

Introduction to big data and its

tools

Hands-on immersion on big

data tools

Big data sources - Web, Social media and text analytics

Advanced big data sources - Mobile phone and other

sensors

Big data courses

Can a statistician become a data

scientist?

The use of R in official statistics:

model based estimates

Time-series econometrics

Methodology courses

Nowcasting

Activity

29 Feb 2 Mar 21 24 Jun

7 10 Nov

12 15 Sep

5 7 Apr 8 10 Jun 24 26 Feb

Eurostat

Actions Contract

2016-2017 (22 months) Selected big data sources

Analysis of legislative situation in EU Review of ethical principles, Elaboration of Communication Strategy See also the Feasibility study on the use of mobile positioning data for tourism

statistics (report on feasibility of access)

Challenges Access to data & continuity of

access Data security & privacy concerns Use of data only for statistical

purposes Impact on the public opinion of

privacy and security concerns ?

17

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics /

Communication

Partnerships

Pilots

Eurostat

Actions Sandbox IT infrastructure for experimenting

UNECE, Eurostat, EU

New Software (Hadoop, Pig, Python, ) Pilots for gaining experience

Challenges Computing capacity, hardware ? Analytical tools, software? Storage ?

18

Policy Quality Skills

Experience sharing Legislation

IT Infrastructures

Methods Ethics /

Communication

Partnerships

Pilots

Eurostat

Thank you for your Attention!

To be honest, Mr Wirthmann, I don't care how many yottabytes you have

Big Data,can you tell me where I want to gofrom how? to deliveringBig data at EurostatModernisation: ESS VisionNmero de diapositiva 5What is the BIGD project? and the ESSnet Big Data?Nmero de diapositiva 8Big Data project ApproachNmero de diapositiva 10Nmero de diapositiva 11PilotsESS Big Data PilotsNmero de diapositiva 14Nmero de diapositiva 15ESTP courses supporting big data (2016)Nmero de diapositiva 17Nmero de diapositiva 18Thank you for your Attention!