big data & real-time campaign_gse - external

7/23/2019 Big Data & Real-time Campaign_GSE - External

1/29

2011 IBM Corporation

Telecom Solutions Lab

BAO & Big Data OverviewApplied to Real-time Campaign

GSE

Joel Viale

Telecom Solutions Lab Solution Architect


2/29



Agenda

2

BAO & Big Data - Overview

Customer use-cases

Live Prototypes:

Streams for Real-time Campaign

Big Insights for Web Log Analysis


3/29



3

The Business Analytics Journey: Transform data into actionableinsight!

AnalyzeIntegrate

Transactional

& CollaborativeApplications

Manage

Business AnalyticsApplications

Internal & External

formation Sources

Cubes

Streams

Big DataMaster

Data

Content

Data

Streaming

Information

DataWarehouses

Govern

QualitySecurity &

PrivacyLifecycle


4/29



4

What is Big Data? How could it impact your organization?

Big Data technologies describe a newgeneration of technologies and architectures,designed to economically extract value fromvery large volumes of a wide variety of data,

by enabling velocity capture, discovery and/oranalysis.

- Matt Eastwood, IDChttp://www.tweetdeck.com/twitter/matteastwood/~hHgsU


5/29



5

The Big Data Challenge Manage and benefit from massive and growing amounts of data

Handle varied data formats (structured, unstructured, semi-

structured) and increased data velocity Exploit BIG Data in a timely and cost effective fashion

Collect Manage

Integrate Analyze

COLLECT MANAGE

INTEGRATE ANALYZE


6/29



6

Bring Together a Large Volume and Variety of Data to Find NewInsights

Identify criminals and threats from

disparate video, audio, and data

feeds

Make risk decisions based on real-time transactional data

Predict weather patterns to planoptimal wind turbine usage, and

optimize capital expenditure on

asset placement

Detect life-threatening

conditions at hospitals in time

to intervene

Leverage Multi-channel customer

sentiment and experience

analysis to upsell in real time


7/29



7

The Solution IBMs Big Data PlatformBring together any data source, at any velocity, to generate insight

Analyzing a variety of data atenormous volumes

Insights on streaming data Large volume structured data

analysis

IBM Big DataPlatform

Variety

Velocity

Volume


8/29



8

2 ways of creating insights: Streaming and Storing Big Data

Real time analysis of data-in-motion

Structured or unstructured

Analytic operations on streaming data in real-time

Data QueriesResults

b) streaming data

Data QueriesResults

b) streaming data

Data QueriesResultsData QueriesResults

b) streaming data

Managing and analyzing Internet-scale volumes

Structured or unstructured data

Based on Googles MapReduce technology

Inspired by Apache Hadoop; compatible with its ecosystem and distribution

Well-suited to batch-oriented, read-intensive applications

Integrated Text Analytics


9/29



9

9

-Distributed File System (HDFS)

-Map/Reduce

How to deal with Big Data: Split a big job into smaller piecesHighly Massive Parallel Processing (MPP)


10/29



10

InfoSphere Streams enables highly scalable stream processing

InfoSphere Streams provides a programming model for defining data flow graphs consisting of data sources

(inputs), operators, and sinks (outputs)

controls for fusing operators into processing elements (PEs) infrastructure to support the composition of scalable stream processing

applications from these components deployment and operation of these applications across distributed x86

processing nodes,

when scaled-up processing is required


11/29



11

Big Data complements traditional warehouse & analyticsinfrastructure

Streams

Internet

Scale

TraditionalWarehouse

In-Motion

Analytics

Data Analytics,

Data Operations

& Model

Building

Results

Internet Scale

Database &

Warehouse

At-Rest Data

Analytics

Results

Ultra Low

Latency Results

InfoSphereBig

Insights

InfoSphereBig

Insights

Moving Beyond the Traditional Warehouse

Traditional /

Relational

Data Sources

Traditional /

Relational

Data Sources

Non-Traditional /

Non-Relational

Data Sources

Non-Traditional /

Non-Relational

Data Sources

Non-Traditional/

Non-RelationalData Sources

Non-Traditional/

Non-RelationalData Sources

Traditional/Relatio

nal Data Sources

Traditional/Relatio

nal Data Sources


12/29



Agenda

12


Customer use-cases

Live Prototypes:




13/29



13

Examples of Streams use-casesStock market Impact of weather on

securities prices Analyze market data at

ultra-low latencies

Fraud prevention Detecting multi-party fraud Real time fraud prevention

e-Science Space weather prediction Detection of transient events

Synchrotron atomic research

Health & Life Sciences Neonatal ICU monitoring Epidemic early warning system Remote healthcare monitoring

Transportation

Intelligent traffic management

Law Enforcement, Defense &Cyber Security

Real-time multimodal surveillance Situational awareness Cyber security detection

Manufacturing Process control for

microchip fabrication

Natural Systems Wildfire management Water management

Telephony CDR processing Social analysis Churn prediction Geomapping

Other Smart Grid Text Analysis Whos Talking to Whom? ERP for Commodities FPGA Acceleration


14/29



Streams in Telecommunications

14

Data in motion

CDRs

Billing

CRM

Location

Account Mgt

Internet

Network

Millions of

events per

second

Microsecond

Latency

Dropped Calls

Outgoing InternationalCallsCall Duration

Extra Call

Contract Expiration

Entered new cell

New Top-Up

5 minutes left on pre-paid

MDM

EDW

Invoice Issued

Campaign MgtReal-time

Promo

Fraud Detection

ServiceAssurance

NetworkMonitoring

Real-timeMonitoring(Voucher

Recharge &Service Usage)

3Drop

pedCallsi

nthela

st

hour

Locatio

n-base

dPromo

5Outg

oinginte

rnation

alCallsinth

elast

day

100 SMSs sent in 1hour500$ top-up on pre-paid

Nogamedownloadinlast30minutes

500FailedSMSDeliverieslast10minutes

AggregatedDataRecordsatService

Level

AggregatedDataRecordsatCell

Level

AggregatedVoucherRecharge

Data Aggregated at

Single Customer

Level

Data Aggregated at

Cell or Service

Level


15/29



CDR cleansing and pre-processing

The Pain Point:

A CSP has 6 Billions CDRs per day!

As much as 500,000 per second peak rate

Kept up, but re-processing meant waiting fornights/weekends to catch up

The Solution:

InfoSphere Streams to clean and pre-processCDRs

De-duplication of CDR against 15 days of data(90B CDRs)

Offloading CDRs processing to Streams platformincreased the performance of their warehouse forother analytics

Single platform for mediation and real timeanalytics reduced IT complexity


16/29



Real-time marketing campaign

InsightInsight

InformationInformation

pre

scriptive

pre

scriptive

DataData activ

e

active

Business

flexibility&r

esponsiveness

Business value

The Pain Point

100M CDRs per day from SMS from25M subscribers, only used to sendbills to customers

The Solution

InfoSphere Streams to create real-time marketing promotions andmonetize the CDRs


17/29



Typical Telco Use-Cases with InfoSphere Streams

ETL-like for massive data to off-load traditional ETL and Warehouse

Mediations

Prediction and prevention of customer churn In-motion analytics can help identify group leaders

Real Time context sensitive promotions Location based promotions

Customer minutes usage based promotions Customer smart phone browsing pattern based promotions Network equipment utilization based promotions

Real Time & pro-active network monitoring

Real Time call traffic & revenue monitoring

SMS spam filtering

Fraud detection17


18/29



Typical Telco Use-Cases with InfoSphere Big Insights

CDR Use Cases:

User behavior analysis

Prediction on the basis of Customer churn

Service association analysis

Social Media Analytics: Get insights, improve campaign, influence opinions

Fraud Detection: Monitoring/Mining transaction logs to detect fraud activities

Recommendation Engines: Improving user experience and likelihood of purchase

Network Traffic Logging: Network optimization & fault detection

Advertising Optimization: in support of advertising based models

Server Logs: Fault detection / Performance related analysis

Email and Email Logs:

Consumer email analysis & decision system

18


19/29



Product Capabilities

Big Insights at the core of Cognos Consumer Insight

BLOGSBLOGS

DISCUSSION FORUMSDISCUSSION FORUMS

TWITTERTWITTER

NEWSGROUPSNEWSGROUPS

FACEBOOKFACEBOOK

Source Areas

Dimensional Analysis

Filtering

Voice

Keyword Search

Dimensional Navigation

Drill Through to Content

Relevant Topics Associated Themes

Ranking and Volume

Relationship Tables Relationship Matrix

Relationship Graph

COMPREHENSIVEANALYSIS

SENTIMENT

EVOLVING TOPICSAFFINITY ANALYTICS

Business Drivers

Customer CareCustomer CareCorporate ReputationCorporate Reputation

Campaign EffectivenessCampaign Effectiveness

Competitive Analysis

Product Insight

MULTILINGUALMULTILINGUAL

19


20/29



Agenda

20


Customer use-cases

Live Prototypes:




21/29



, &

Real-time Campaign Solution Architecture

InfoSphere Streams

Unica Suite

SPSS

Cognos Suite

Netezza


22/29



Outbound campaign, enhanced with Real-time Monitoring andadvanced predictive analytics

Events Collaboration Content MonitoringAnalytics Rules

ProductManager

MarketingManager

VP ofMarketin

g

Customer

DefineMarketingStrategy

CreateCampaig

n

IdentifyTargetCustomers

andOffering

ApproveCampaig

n

Monitor &EvaluateCampaig

n

ExecuteCampaig

n

DeliveryChannel

Real-time Monitoring

Detect situations andpatterns in service usage.Determine the need of acampaign

Predictive Analytics

Segment Customers basedon Churn Prediction,likelihood of campaignresponse, LTCV, etc

Presence & Rules based channel

selection

Send campaign message on appropriatechannel (including social networks),

based on customer preferences andavailability/presence

Monitor key KPIs

Sales/Usage increase % of campaign response

22

Collectand

aggregatexDRs

Network

Real-time xDRs

Processing

Collect, correlate andaggregate data records inreal-time


23/29



23

Real-time campaign, based on network events

Manage

CampaignDelivery

MonitorCampaignExecution

Event Based

CampaignProcessing

Evaluate

CampaignEffectiveness

Real-time Event Detection

InfoSphere Streams correlatemultiple data records in real-time down to the customerlevel.Here: detects a number ofdropped calls in a certaintime

Event-based Campaign

The event-based campaignprocessing is triggered: acampaign is executed in Unica

Campaign.

Message Delivery

The campaign message isdelivered to the customeron the appropriate channel

Campaign monitoring

Monitor campaign executionand analyze effectiveness

CDRs

CDR

Feed

Dropped CallsFiltering

Dropped CallsAggregation

Per Subscriber

InfoSphere StreamsReal-time event detection


24/29



Agenda

24


Customer use-cases

Live Prototypes:




25/29



The Sample Outdoors Companyhas many visitors a year to their

website and only a subsetresulted in purchases. Theywould like to know what is

happening to the other visits?

25


26/29



Every time the user browses a page, the click-stream data is logged

User goes to

web site

Additem

to cart

Searchfor

item

Userchecks

out

26


27/29



Web logs capture click streams of user

Basic analysis of web logs can provide valuable insights to web site usage

and user behaviors How many users going to my web site?

What pages and products are people looking at?

Additional sessionization and aggregation analysis help to detect valuableclick patterns and provide insights for:

Improving customer service

Ads promotion and sales improvement

Detection of incidents and errors

Click Stream Data Analysis for Customer Insights

27


28/29



Big Insights Web Log Analysis Solution Overview

Web ServerLogs

BigInsights Analytics

28


29/29



Solution Architecture for Web Log Analysis

(, )

()

29

big data & real-time campaign_gse - external

Documents