customer event hub – a modern customer 360° view with datastax enterprise (dse)

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH

Customer Event HubCustomer 360° view with DataStax Enterprise (DSE)

Guido Schmutz – 4.7.2017

@gschmutz guidoschmutz.wordpress.com

Guido Schmutz

Working at Trivadis for more than 20 yearsOracle ACE Director for Fusion Middleware and SOAConsultant, Trainer Software Architect for Java, Oracle, SOA andBig Data / Fast DataHead of Trivadis Architecture BoardTechnology Manager @ Trivadis

More than 30 years of software development experience

Contact: guido.schmutz@trivadis.comBlog: http://guidoschmutz.wordpress.comSlideshare: http://www.slideshare.net/gschmutzTwitter: gschmutz

2

COPENHAGEN

MUNICH

LAUSANNEBERN

ZURICHBRUGG

GENEVA

HAMBURG

DÜSSELDORF

FRANKFURT

STUTTGART

FREIBURG

BASEL

VIENNA

With over 600 specialists and IT experts in your region.

14 Trivadis branches and more than600 employees

200 Service Level Agreements

Over 4,000 training participants

Research and development budget:CHF 5.0 million

Financially self-supporting andsustainably profitable

Experience from more than 1,900 projects per year at over 800customers

3

Agenda

4

1. Customer 360° View – Introduction2. Customer 360° View – Challenges3. Customer 360° View – DataStax Enterprise (Graph) to the rescue

Customer 360° View - Introduction

5

Why Customer 360° View?

“Get closer than ever to your

customers. So close that you tell

them what they need well before

they realize it themselves.”

Steve Jobs, Apple

q Enhance customer service

q Provides real-time personalization

q Opens doors to new applications

q Increases security

q Lowers operational costs

q Increases operational efficiency

6

Customer 360°: Experience Expectations

7

“The Amazon effect” – why can’t I do .... as easy as buying a product on Amazon

8

Each time a customer is exposed to an improved digital experience, their engagement expectations are reset to a new higher level.

Source: Forrester

Customer 360°: Experience Expectations

Consistent

Acrossallchannels,brandsanddevices

Personalized

Toreflectpreferencesandaspiration

Relevant

Inthemomenttocustomer’sneedsandexpectations

Contextualized

Topresentlocationandcircumstances

9

Customer 360 – Key Use Cases

• Customer micro segmentation

• Next Best Offer• Campaign Analytics• Geo-Location

Analytics• Recommendation

Models

• Churn Modeling & Prediction

• Rotational / Social Churn

• Customer Lifetime Value

• Sentiment Analytics• Price Elasticity

Modeling

• Proactive Care Dashboard

• Customer Lifetime Value

• Subscriber Analytics• QoS Analytics• Real-Time Alerts

Target Marketing & Personalization

Churn Prevention & Customer Retention Proactive Care

10

Customer 360° - What do I need to know?

Who are you?

Where are you?

What have you purchased?

What content do you prefer?

Who do you know?

What can you afford?

What is your value to the business?

How / why have you contacted us?

11

From Static to Dynamic, Real-Time Micro-Segmentation

AgeGenderAverage SpendPrice PlansUsage HistoryData, Voice, TextBilling HistoryDevice Upgrade

Traditional Segmentation

AgeGenderAverage SpendPrice PlansUsage HistoryData, Voice, TextBilling HistoryDevice Upgrade

Device HistoryOther products / servicesBundling preferencesOffer HistoryCampaign Adoption HistoryCall Center Tickets

LocationSocial Influence

Applications UsedContent Preferences

Usage DetailsRoaming Analysis

Travel Patterns

QoS HistoryHousehold Analysis

Lifetime ValueChurn Score

Clickstream InfoChannel Preferences

Survey

Real-Time Micro-Segmentation

12

From Static to Dynamic, Real-Time Micro-Segmentation

AgeGenderAverage SpendPrice PlansUsage HistoryData, Voice, TextBilling HistoryDevice Upgrade

Traditional Segmentation

AgeGenderAverage SpendPrice PlansUsage HistoryData, Voice, TextBilling HistoryDevice Upgrade

Device HistoryOther products / servicesBundling preferencesOffer HistoryCampaign Adoption HistoryCall Center Tickets

LocationSocial Influence

Applications UsedContent Preferences

Usage DetailsRoaming Analysis

Travel Patterns

QoS HistoryHousehold Analysis

Lifetime ValueChurn Score

Clickstream InfoChannel Preferences

Survey

Real-Time Micro-Segmentation

13

IndividualizationEngaging customers as a segment of one in real-time by listening, capturing, measuring, assessing, and addressing intent across every enterprise touchpoint.

Source: Forrester

Customer 360° View - Challenges

14

Customer 360° View - Traditional Flow Diagram

Enterprise Data Warehouse

ETL / Stored Procedures

Data Marts / AggregationsLocation

Social

Clickstream

Segmentation & ChurnAnalysis

BI Tools

Marketing Offers

Billing &Ordering

CRM / Profile

MarketingCampaigns

15

Customer 360° View - Traditional Flow Diagram

Enterprise Data Warehouse

ETL / Stored Procedures

Data Marts / AggregationsLocation

Social

Clickstream

Segmentation & ChurnAnalysis

BI Tools

Marketing Offers

Billing &Ordering

CRM / Profile

MarketingCampaigns

Limited Processing

Power

Does not model easily to traditional

database schema

Limited Processing

Power

Storage Scaling

very expensive

Based on sample /

limited dataLoss in Fidelity

Other / New Data Sources

High Voume

and Velocity

16

Key Challenges in Driving a Customer 360° ViewData Silos

New Data Sources Costs of Data Processing

Data Volumes

• Multiple Data Silos• Often store overlapping and

conflicting information

• Data growing rapidly• Internet of Things will add to

that substantially

• Semi/Un-Structured Data Sources

• Streaming / Real-time data• Critical for building a true

360° view

• Cost prohibitive• Cost of storing data in

relational database systems per year

Clickstream Location/GPS

Call centerRecords

Social Media

17

Customer 360° View: Why status quo won’t work?

• static version of the customer profile in their data warehouse• Mainly structured data• Only internal data• Only “important” data• Only limited history

• Activity data – clickstream data, content preferences, customer care logs are kept in siloes or not kept at all

Data Analyst

Data Analyst

Data Analyst

Data Analyst

Data Analyst

DetailedCustomerActivityDatasitsinsilos!

18

Customer Journey through multiple “Siloed Systems”

19

SocialMedia

CallCenter

ComplaintManagement

Marketing

Coupons

Warehouse CRM

Shipping

Billing

OrderProcessing

Web Application

Customer Journey through multiple “Siloed Systems”

20

SocialMedia

CallCenter

ComplaintManagement

Marketing

Coupons

Warehouse CRM

Shipping

Billing

OrderProcessing

Web Application

Customer 360° View – DataStaxEnterprise (Graph) to the rescue

21

Why using Graph for Customer 360° View

22

Traditional RDBMS• Multiple Data Locations => siloes• Not all information related• difficult to access all information and relate

to each other

Graph Database• Connect all customer-related information in

one central place• Support for analytics on graph• Performance & Scalability

PartnerIdFirstNameLastNameBirthDateProfileImageURL

IdCustomerIdWhenWhereTypeDescription

Contacts

AddressIdPartnerIdStreetStreetNrZipCodeCityCountry

IdNamePriceDescriptionImageURL

Products

CustomerIdProductId

Order

IdCustomerIdWhenWhereInteractionTypeDescription

Customer Service

IdProductIdCommentUserId

Product Reviews

IdPartnerIdTwitterIdFacebookId

Customer

Geo Point

Customer

Address

Product

ownerOf (since)

interestedIn (since, degree)

lives (since)

ActivityparticipatesIn Termuses

uses

Employee

interactsWith

reference

idname

idwhenwheretext

idfirstNamelastNamebirthDateprofileImageURL

usesidnameimageURL

idstreetstreetNrzipCodecity

idnamealternateNameslocationpopulationelevation

Country

neighbour

belongsTo

idnameiso2iso3isoNumericcapitalareapopulationcontinent

uses

ActivityType

idname

belongsTo

idname

at

knows(since)

mentions

uses

Customer 360° View - Example

23

Geo Point

Customer

Address

Product

ownerOf (since)

interestedIn (since, degree)

lives (since)

ActivityparticipatesIn Termuses

uses

Employee

interactsWith

reference

idname

idwhenwheretext

idfirstNamelastNamebirthDateprofileImageURL

usesidnameimageURL

idstreetstreetNrzipCodecity

idnamealternateNameslocationpopulationelevation

Country

neighbour

belongsTo

idnameiso2iso3isoNumericcapitalareapopulationcontinent

uses

ActivityType

idname

belongsTo

idname

at

knows(since)

mentions

uses

Customer 360° View - Example

24

Hadoop ClusterdHadoop ClusterDSE Cluster

Batch Data Ingestion into Customer Hub

Billing &Ordering

CRM / Profile

MarketingCampaigns

Location

Social

Clickstream

Sensor Data

WeatherData

MobileApps

Emails

v

File Import /SQL Import

Cassandra DSE Graph

25

highlatency

Event Stream

Batch Data Ingestion into Customer Hub

DSEGraphDSE

GraphLoaderRDBMS

GroovyScript

ClickStream

26

Geo Point

Customer

Address

Product

ownerOf (since)

interestedIn (since, degree)

lives (since)

ActivityparticipatesIn Termuses

uses

Employee

interactsWith

reference

idname

idwhenwheretext

idfirstNamelastNamebirthDateprofileImageURL

uses

idnameimageURL

idstreetstreetNrzipCodecity

idnamealternateNameslocationpopulationelevation

Country

neighbour

belongsTo

idnameiso2iso3isoNumericcapitalareapopulationcontinent

uses

ActivityType

idname

belongsTo

idname

at

knows(since)

Transformation CSV/JSON

PartnerIdFirstNameLastNameBirthDateProfileImageURL

IdCustomerIdWhenWhereTypeDescription

Contacts

AddressIdPartnerIdStreetStreetNrZipCodeCityCountry

IdNamePriceDescriptionImageURL

Products

CustomerIdProductId

Order

IdCustomerIdWhenWhereInteractionTypeDescription

Customer Service

IdProductIdCommentUserId

Product Reviews

IdPartnerIdTwitterIdFacebookId

Customer

Real-Time Insights?

Real-Time Data Ingestion into Customer Event Hub

Microservice Cluster

Microservice State

{}

API

Stream Processing Cluster

StreamProcessor

State

{}

API

Billing &Ordering

CRM / Profile

MarketingCampaigns

Location

Social

Clickstream

Sensor Data

WeatherData

MobileApps

Email

File Import / SQL Import

EventStream

Event Hub

Event Hub

Event Hub

EventStream

EventStream

Hadoop ClusterdHadoop ClusterDSE Cluster

Cassandra DSE Graph

27lowlatency

Process native Event Streams

Twitter

Tweet-to-Cassandra

Wide-Row/Search

Tweet

Graph

Tweet-to-Graph

CustomerReference

ClickStream ClickStream

Activity-to-Graph

DSECluster

28

Geo Point

Customer

Address

Product

ownerOf (since)

interestedIn (since, degree)

lives (since)

ActivityparticipatesIn Termuses

uses

Employee

interactsWith

reference

idname

idwhenwheretext

idfirstNamelastNamebirthDateprofileImageURL

uses

idnameimageURL

idstreetstreetNrzipCodecity

idnamealternateNameslocationpopulationelevation

Country

neighbour

belongsTo

idnameiso2iso3isoNumericcapitalareapopulationcontinent

uses

ActivityType

idname

belongsTo

idname

at

knows(since)

Event Hub Streaming AnalyticsSensor Storage

Single vs. Batch Insert

29

• One Event ends up in many modifications of vertex and edges

• many round-trips needed if done with single API calls

• Single “batched” API call using Groovy script provides 3 – 5x performance gains

DSEGraphActivity-to-GraphTweetUser

publishesTerm*uses*

DSEGraphActivity-to-Graph

…...

Geo Point

Customer

Address

Product

ownerOf (since)

interestedIn (since, degree)

lives (since)

ActivityparticipatesIn Termuses

uses

Employee

interactsWith

reference

idname

idwhenwheretext

idfirstNamelastNamebirthDateprofileImageURL

uses

idnameimageURL

idstreetstreetNrzipCodecity

idnamealternateNameslocationpopulationelevation

Country

neighbour

belongsTo

idnameiso2iso3isoNumericcapitalareapopulationcontinent

uses

ActivityType

idname

belongsTo

idname

at

knows(since)

Process CDC (Change Data Capture) Events

CDC

Customer-to-Cassandra

Wide-Row/Search

CustomerEvent

GraphCustomer-to-

Graph

CustomerCDC

AddressCDC

ContactCDC

Aggregate-to-Customer

DSECluster

30

Geo Point

Customer

Address

Product

ownerOf (since)

interestedIn (since, degree)

lives (since)

ActivityparticipatesIn Termuses

uses

Employee

interactsWith

reference

idname

idwhenwheretext

idfirstNamelastNamebirthDateprofileImageURL

uses

idnameimageURL

idstreetstreetNrzipCodecity

idnamealternateNameslocationpopulationelevation

Country

neighbour

belongsTo

idnameiso2iso3isoNumericcapitalareapopulationcontinent

uses

ActivityType

idname

belongsTo

idname

at

knows(since)

PartnerIdFirstNameLastNameBirthDateProfileImageURL

IdCustomerIdWhenWhereTypeDescription

Contacts

AddressIdPartnerIdStreetStreetNrZipCodeCityCountry

IdNamePriceDescriptionImageURL

Products

CustomerIdProductId

Order

IdCustomerIdWhenWhereInteractionTypeDescription

Customer Service

IdProductIdCommentUserId

Product Reviews

IdPartnerIdTwitterIdFacebookId

Customer

Event Hub Streaming AnalyticsSensor Storage

Real-Time Data Ingestion into Customer Event Hub

Microservice Cluster

Microservice State

{}

API

Stream Processing Cluster

StreamProcessor

State

{}

API

Billing &Ordering

CRM / Profile

MarketingCampaigns

Location

Social

Clickstream

Sensor Data

WeatherData

MobileApps

Email

File Import / SQL Import

EventStream

Event Hub

Event Hub

Event Hub

EventStream

EventStream

Hadoop ClusterdHadoop ClusterDSE Cluster

Cassandra DSE Graph

31

How to implement an Event Hub? Apache Kafka to the rescue

• publish-subscribe messaging system

• Designed for processing high-volume, real time activity stream data (logs, metrics, social media, …)

• Stateless (passive) architecture, offset-based consumption

• Initially developed at LinkedIn, now part of Apache

• Peak Load on single cluster: 2 million messages/sec, 4.7 Gigabits/sec inbound, 15 Gigabits/sec outbound

Reliable Data Ingestion in Big Data/IoT

Kafka Cluster

Consumer Consumer Consumer

Producer Producer Producer

Broker 1 Broker 2 Broker 3

ZookeeperEnsemble

Apache Kafka Connect

• Scalably and reliably streaming data between Apache Kafka and other data systems

• not an ETL framework• Pre-build connectors available for Data

Source and Data Sinks• JDBC (Source)• Cassandra (Source & Sink)• Oracle GoldenGate (Source)• MQTT (Source)• HDFS (Sink)• Elasticsearch (Sink)• MongoDB (Sink)

Reliable Data Ingestion in Big Data/IoT

Source: Confluent

Declarative Dataflow Definition & Execution

34

Apache NiFi StreamSets

Hadoop ClusterdHadoop ClusterDSE Cluster

Real-Time Data Ingestion into Customer Event Hub

OLA

P

Microservice Cluster

Microservice State

{}

API

File Import / SQL Import

OLT

P

ParallelProcessing

Cassandra

Cassandra

StreamProcessing

DSEFS

Billing &Ordering

CRM / Profile

MarketingCampaigns

Location

Social

Clickstream

Sensor Data

WeatherData

MobileApps

Email

EventStream

Event Hub

Event Hub

Event Hub

EventStream

CassandraReplication

35

Hadoop ClusterdHadoop ClusterDSE Cluster

Real-Time Data Ingestion into Customer Event Hub

OLA

P

Microservice Cluster

Microservice State

{}

API

File Import / SQL Import

OLT

P

ParallelProcessing

Cassandra

Cassandra

StreamProcessing

DSEFS

Billing &Ordering

CRM / Profile

MarketingCampaigns

Location

Social

Clickstream

Sensor Data

WeatherData

MobileApps

Email

EventStream

Event Hub

Event Hub

Event Hub

EventStream

CassandraReplication

36

Questions which can only be answered by Graph

37

Dependencies•Failurechains•Orderofoperation

Matching/CategorizingHighlightvariantofdependencies

ClusteringFindingthingscloselyrelatedtoeachother(friends,fraud)

Flow/CostFinddistributionproblems,efficiencies

SimilaritySimilarpathsorpatterns

Centrality,SearchWhichnodesarethemostconnectedorrelevant

Source: Expero

Questions which can only be answered by Graph -Visualize Customer 360

38

Source: Expero

Questions which can only be answered by Graph -Visualize Customer 360

39

KeyLines

Cytoscape

LinkuriousJS

SigmajsD3

Source: Expero

Questions which can only be answered by Graph -Visualize Customer 360

40

Keylines: http://keylines.com• powerful JavaScript toolkit for graph visualization• powerful visualizations features: Layouting, Filtering,

Combining & Grouping, Time Bar, Geospatial, …• compatible with any modern browser & any graph

database

Guido SchmutzTechnology Manager

guido.schmutz@trivadis.com

@gschmutz guidoschmutz.wordpress.com