hadoop world 2011: changing company culture with hadoop - amy o'connor, nokia

Using Hadoop to

Change Company Culture

Amy O’Connor

Senior Director, Analytics

1

2

The Amazing Everyday

The Amazing Everyday

3

NOKIA’S HISTORY: 1865 TO Now

4

Data is our newest

raw material

5

US Government data

suggests worsening

conditions in urban

areas…this year

average commute time

has risen by 9 minutes.

TRAFFIC

Federal Highway

Commission, Urban

Congestion Report

January 2011 to

March 2011

6

Animate..? Something to just

make more visually appealing

Note deleted google map

TRAFFIC

IHS Automotive,

October 2011

Global car sales are

growing: there will

be 50% more cars

sold in 10 years as

there are today.

7

More data usually beats

better algorithms Anand Rajaramann

8

• Image Sensors

• Accelerometers

• Gyroscopes

• Compasses

• Pressure Sensors

• Microphones

• Light Sensors

• Assisted GPS

Probe points

collected

monthly from

Nokia alone

Devices in

use around

the world

9

10

80M

18M 24

Data Storage & Analysis Landscape

11

Data Silos

12

Traffic Probes

Ad Data

Search Logs

Places Registry

Consumer Profile

Device Data

Smart Data

Combining

sets of

behavioral &

contextual data

13

14

A Good Way to Change Corporate Culture

Getting Children to Eat Peas… Tell them you expect them to eat their peas.

Reward them with ice cream if they did.

Explain why it’s good for them to eat their peas.

Eat your own peas as a good role model.

Leann Lipps Birch, Head of Human Development & Family Studies Pennsylvania State University

15

Getting Children to Eat Peas…

Leann Lipps Birch, Head of Human Development & Family Studies Pennsylvania State University

16

Put them with children who love peas.

Change the stories they tell.

Nokia Data

Asset

Search

Log Files

Points of Interest

Advertising

Ad Inventory

Ad Canvas

Campaign Promotions Location

Premium Content

Favorite Routes

Log Files

Map Tiles

Imagery

Social

Social UGC

Universal Share

Journeys Data

Event Info

Navteq

All Probes Data

3D Imagery

Street View

Feature Recognition

Identity

Consumer Profile

SSO

Device Activation

Campaigns

Contacts

Device Programs

NAC, IIA

Windows Phone

Panel data

Equipment Master

Factory Data

Care & Marketing

Nokia IT

Registrations

Device Updates

Media

Products, Transactions

Songs, Delivery

Device/User CRM

Activation Info

Net Promotor

PC Suite

17

18

Users

Domain Expertise, Computer Science

Developers/ Applications

Domain Expertise, Statistical Skills,

Computer Science

Data Scientists

Domain Expertise, Statistical Skills

Data Analysts

Domain Expertise

Decision- Makers

Map Reduce

Pig Hive Oozie

TeraData Key Value

Store

Predictive Analytics

Dashboards Offline

Analysis

HBase

HBase

HBase

FTP

Scribe

Flume

HDFS

Analytics

19

Collaborative Working Model

Load

Transform

Extract

Platform

Present

Analyze and

Aggregate

Map Reduce

Pig Hive Oozie

TeraData Key Value

Store

Predictive Analytics

Dashboards Offline

Analysis

HBase

HBase

HBase

FTP

Scribe

Flume

HDFS

Analytics

20

Load

Transform

Extract

Platform

Present

Analyze and

Aggregate

Map Reduce

Pig Hive Oozie

TeraData Key Value

Store

Predictive Analytics

Dashboards Offline

Analysis

HBase

HBase

HBase

FTP

Scribe

Flume

HDFS

Analytics

Collaborative Working Model

Co-located developer clusters, pre-production cluster, product cluster

Create Hive

Schema

User Metadata Interfaces

Metadata Catalog

Data Model

Definition

Develop Validation

Job

Develop Cleanse

Job

Monitor and

Manage Transform

Create Library of MR Jobs

Develop Partition

Job

Catalog Data

Sources

Define Custom

ETL

Define Standard

ETL

Monitor and

Manage Data Feed

Integrate Streaming

Data

Integrate Historical

Data

To DataOS,

Structured Data

BI Tools: SPSS,

Tableau, Cognos

Customer Dashboard

AD Hoc Reporting

Hive QL and Pig Queries

MR Agg Job and Data to Oracle

MR Analysis

Job

Rec, Engines

Mahout Machine Learning

Data OS Product Teams and/or DataOS

MR Agg Job and Data to

Teradata

21

Smart Data: Behavioral & Contextual

Map Reduce

Pig Hive Oozie

TeraData Key Value

Store

Predictive Analytics

Dashboards Offline

Analysis

HBase

HBase

HBase

FTP

Scribe

Flume

HDFS

Analytics

Local Search

NAC

Ad Router

Standard ETL

• Clean data, remove bad record and health checks

• Partition data by date, hour, type

• Archive raw data

Aggregation

• Merge multiple data sources

• Implement app logic (e.g.; round up latitude/longitude to 3 decimals)

Aggregation

• Update top searches table and geo activity table in Oracle

22

Merchant Portal Heatmap

23

Mapping the World

© 2011 Nokia Company Confidential

23

24

Mapping the World

© 2011 Nokia Company Confidential

Probe data indicates the location, speed, heading, time etc. about a mobile device.

Billions of probe records per week.

24

Covers almost the entire world.

25

100% AGR

Probe Density: Urban and Arterial

26

792527719 (Jackson Blvd/Financial Pl)

From Ref, One lane

27

24 Hours in Our Analytics Ecosystem

~2TB ingested

350M messages via scribe

>3000 MR jobs

10TB processed

28

Spreading the Word

Smart Data Newsletter Stories

Realtime Dashboards

Data Asset Catalog

29

Thanks!

30

The Amazing Everyday