big data overview

BIG DATAArnon Rotem-Gal-Oz

Director of Technology Research, AmdocsThe blind men and the elephant. Poem by John Godfrey Saxe (Cartoon originally copyrighted by the authors; G. Renee Guzlas, artists http://www.nature.com/ki/journal/v62/n5/fig_tab/4493262f1.html

1880 US Census

HollerithTabulating Machine

Hollerith photos by Martin Wichary : http://www.flickr.com/photos/mwichary/4358926764/in/photostream/

Source: Silicon Angle http://siliconangle.com/blog/2013/11/13/how-big-is-big-data-really/

Big data happens when the data you have to process is bigger than what you can process in the given time with current

technologies

Myth: Big data = keep all data

Source: Big Data Public Private Forum : http://www.big-project.eu/sites/default/files/D2.2.1_First%20draft%20of%20Technical%20white%20papers_FINAL_v1.01_0.pdf

Source: Big Data Public Private Forum : http://www.big-project.eu/sites/default/files/D2.2.1_First%20draft%20of%20Technical%20white%20papers_FINAL_v1.01_0.pdf

Some Telco Numbers

Source: Wikipediahttp://upload.wikimedia.org/wikipedia/commons/5/50/Telephone_operators,_1952.jpg

So, what do we do with all this data?

Source: Wikipedia http://upload.wikimedia.org/wikipedia/commons/0/06/UPS_Truck.jpg

It’s the insights, stupid*

* With apologies to Bill Clinton

Source: Silicon Angle http://siliconangle.com/blog/2013/11/13/how-big-is-big-data-really/

Big data analytics is when sample = N

• Big data happens when the data you have to process is bigger than what you can process in the given time with current technologies

“My daughter got this in the mail!, She’s still in high school,

and you’re sending her coupons for baby clothes and cribs? Are you trying to encourage her to

get pregnant?”

• Source: Forbes http://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did/

We need to watch out thatAnalytics won’t get too creepy

When people hear big data they think

fast data

Source: Steve Jones Cap Geminihttp://www.no.capgemini.com/node/778541

Subscribers

Collect& Filter Correlate

(simplified) Network proactive care flow

Account

Event Store

Identify & Predict NetworkFailures

ReimburseVIPs

Prioritize technicians

Identify impact on

high valued Accounts

Source: Silicon Angle http://siliconangle.com/blog/2013/11/13/how-big-is-big-data-really/

Big data is when we can handle data fast enough to make a difference

• Big data happens when the data you have to process is bigger than what you can process in the given time with current technologies

• Big data analytics is when sample = N

Technology space

The Elephant in the room

Hadoop Stack

Map/Reduce

HDFS

HBase

PigHive

ZooKeeper

Oozie MahoutGiraph

Schema on read

Move data to computation

Maybe we should rethink moving data to computation…

Source : http://my-inner-voice.blogspot.co.il/2012/06/haddop-101-paper-by-miha-ahronovitz-and.html

Map/reduce

Source: http://www.bodhtree.com/blog/2012/10/18/ever-wondered-what-happens-between-map-and-reduce/

Customer Segmentation

First name

Last name

ARPU Age Device Country …

Mr. Smith 100 22 iPhone 5s,White USA

John Doe 87 42 Samsung Galaxy S5,Gold France

Lady In Red 105 21 Samsung Note 3, White UK

…

Uluru, Australia by Stuart Edwards (cc) http://en.wikipedia.org/wiki/Uluru#mediaviewer/File:Uluru_Panorama.jpg

K-Means

ARPU

Age

Source : http://pypr.sourceforge.net/kmeans.html

K=3AR

PU

Age

ARPU

Age

Source : http://pypr.sourceforge.net/kmeans.html

New paradigms

Map/Reduce

HDFS

HBase

PigHive

ZooKeeper

Oozie MahoutGiraph

New Paradigms

Map/Reduce

HDFS

HBase

Pig HiveZoo

Keeper

Oozie Mahout

YARN

Giraph

New Paradigms

Map/Reduce

HDFS

HBase

Pig HiveZoo

Keeper

Oozie Mahout

YARN

Giraph SparkStorm

Slider

Flink

Impala

Tez

Presto

Amdocs Analytics & Data Management Heritage

2013

• Proactive Care• TerraScale• Network optimization

• Real time analytics platform

• Single product catalog

• BSS–OSS Integration

• CRM-Billing Integration

OSSAnalytics Platform,

16 Analytics Patents

• aLDM logical data model

• Policy control

Network AnalyticsCRM

2000 2008

Acqu

isiti

ons

Portf

olio

34Information Security Level 2 – Sensitive© 2014 – Proprietary and Confidential Information of Amdocs

Touchpoints & Applications

CRM Self Service E-MailPCRF SMS OtherWi-Fi OffloadCampaign Mng. • • • • • • •

Operational Envelope & Platform Administration

• Security Management

• Configuration Management

• Services Inventory

• Performance Management

• Fault Management

• LoggerCollect & Ingest

Transform & Enrich

Aggregate & Correlate

Drive Insight

Close the Loop

Machine Learn &

ScoreApplication-Ready Data and Analytics/ML Insights

Entities and Profiles

Detailed Data

OSSProbes Social RAN Inventory Usage &

ChargingCRM

Real-Time & Batch Connectors

Insight Platform

Marketing

AnalyticalApplication Framework:

Dashboards & Visualisation

Decisioning Engine

Dynamic Micro Segmentation

Network Care Operations

Source: Silicon Angle http://siliconangle.com/blog/2013/11/13/how-big-is-big-data-really/

• Big data happens when the data you have to process is bigger than what you can process in the given time with current technologies

• Big data analytics is when sample = N

• Big data is when we can handle data fast enough to make a difference

Additional takeaways

• CSPs have always been in the big data business – they just didn’t know it

• Big data is not a panacea • Hadoop is shaping up as the big data OS– Though there are alternatives arriving from the

cloud arena (mesos, kubernetes)

What we covered here is not even

the tip of the iceberg

Source: wikimedia http://commons.wikimedia.org/wiki/File:Iceberg.jpg

Arnon Rotem-Gal-Oz Director of Technology Research, Amdocsarnonrot@amdocs.com / arnon@rgoarchitects.com