lambda architecture 2.0 convergence between real-time analytics, context-awareness and online...
TRANSCRIPT
HUAWEI TECHNOLOGIES CO., LTD.
CEP & PME For Real-Time
Analytics
Big Data Technologies conference December 2014
Sabri SKHIRI – Head of the R&D Architecture Dpt., European Research Center
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Agenda
1. Introduction
2. What is the Lambda Architecture & Lambda 2.0 Proposal with PME
3. Examples of Use cases with the PME
4. Conclusion
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Sabri SKHIRI Head of R&D Architecture – Expert in Distributed system architecture, Soft Engineering & Data analytics
Scientific Publications (selected) A Distributed Data Mining Framework Accelerated with Graphics Processing Units. 2013 IEEE Conference on Cloud Computing and Big Data Large Graph Mining: Recent Developments, Challenges and Potential Solutions. 2012 Lecture Notes in Business Information Processing, Springer AROM: Processing Big Data with DataFlow Graphs and Functional Programming. 2012 IEEE conference on Cloud computing technology and science And 7 others.
Committer on Big Data open source projects launched @ EURA NOVA
R&D Projects at Huawei (selected)
Distributed Graph Storage & Traversal Graph Mining Libs (influence Mngt) Distributed in-memory Machine Learning Platform Pattern Matching Engine / CEP Stream Processing On Going R&D Projects Feature Engine (Deep Learning) Automatic Modeling Data PaaS
R&D Director & Co-founder of EURA NOVA BE, Private research company
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Before Starting - The Big Data ambiguity Demystify the IT Vendor confusion about Big Data – “Is big data == Hadoop?”
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Introduction Real-Time Analytics
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
IT Framework Based on Big Data
Data Value Openness
Leveraging Big Data in IT operations
Driving the Enterprise
operations through Data-
Driven / Knowledge-Driven
processes
Exploration on the big data ecosystem
Value chain from data sources to
insight consumers
Data-Driven Operation
Big Data Requirement Types of Telecom Carriers Main Directions
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Real-time marketing Context-aware applications
Real-time context-aware QoS management
Intelligent Business Process Management (iBPM)
Trends analysis
Marketing automation
Intelligent Business Operations
Dynamic QoE Management
Proactive CEM
Network Management
Shift in Directions New Business models involve moving to Real-Time
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Shift in Directions – Impact on the Technologies Moving from Batch to context-aware & Real-Time Reaction
Value of Data
Data Stored
Event
Info Delivered
Action
time
ETL ETL HD
FS
Hadoop
M/R
ETL ETL ETL
ETL ETL
DB
ETL ETL DWH
Acquire Organize Analyze
Analysi
s
applian
ce
Decide
Minutes/Hours/Days Sub-second
Detect & Decide
In-memory Analysis Accelerator
(On-line, incremental)
Complentary to Hadoop STACK
Context detection &
reaction
Data value over time
Systems must detect the “key”
information as they happen. The value
of the action decreases over time!
The Architecture shift
We are moving from a batch processing architecture to Real-
Time analysis and Context-aware infrastructure. The Hadoop-
based infrastructure is not well suited to this paradigm shift, but
can be used in a complementary way.
X
X
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Matching Pattern among event Streams Going beyond traditional CEP
Matching complex patterns We can match complex definitions of situation including temporal relationships, event correlation and data from application (CRM, network management, HLR, etc.). We studied 20 Business cases and defined why traditional CEP engines cannot express them efficiently.
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
When should we use the Context? Leveraging the context-awareness
When we need to identify “interesting situations” If an application must be able to detect specific situations .Typical UC in Event-Based Marketing.
When we need to react to the context in RT If an application requires to answer to a specific context by taking an action, we need to define what is this context. The SmartSwitch can define such context. For instance: My flight is at 7:00PM, and it is delayed of 30m, and there is a Traffic Jam on the way from my location and the airport=> The service recommends me to leave my meeting in 10m. Typical UC in Marketing & Network.
When we need to correlate information from different systems or apps If we have to set up constraints on different events coming from different systems or applications without being able to touch at those systems, you can express all those constraints between events as correlations and even detecting all events that do no fit the pattern. Typical UC in Business Enablement Cube, Network and QoS.
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Requirement Summary What is needed to support those new kind of RT Analytics Cases?
Expressing a context or a situation by mean of events We need to be able to express a situation by describing all the events that participate to this context and by defining the temporal relationships between them. Once we have defined this context definition (we call it pattern) we must be able to deploy this pattern at run-time, to listen all the involved events and to detect when this pattern occurs. This detection must done through different streams of millions of events per second.
What exist today to express and recognize those patterns? We have 3 existing technologies for in this area: 1. Event Stream Processing 2. Complex Event Processing (CEP) 3. Pattern Matching engine And integrating RT Contexts with
Predictive Models! We need to be able to associate a situation to a process of actions in which we can leverage all the batch-calculated models. But also … Correcting the model by a Feedback Loop Incrementally update the model
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
(Sybase)
Description and differences among those 3 Technos Pattern Matching is the only one that fits our requirements
Stream Processing (Usually) Distributed event processing infrastructure. Give the ability to define the Stream Processing as a DFG (aka Topology). No temporal support, no DSL (CQL), no support for Event correlation.
CEP Based on Stream processing Technology, the CEP exposes a language derived from SQL for event Processing. The objective is to compute KPIs over streams and to define patterns as Thresholds on those KPIs. The language is tuned for defining operations on Streams.
Pattern Matching Engine The objective is NOT to compute operations on streams but well to express complex temporal correlations between Event streams to define a context. The language is then not oriented to operations but well on expressing complex constraints.
[Tatbul 2006]
Storm + CEP Toolkit
http://esper.codehaus.org
Cayuga
AMIT
T-Rex/TESLA
CRS Network Monitoring
PME
at least 5mins 6mins
at most 10mins
We need this kind of technology to describe context
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
The SmartSwitch PME Beyond traditional CEP
Matching complex patterns beyond CEP We can match complex definitions of situation including
temporal relationships, event correlation and data from
application (CRM, network management, HLR, etc.).
Pattern Query
Language
Algebraic
Transformation
Distributed share-nothing
deployment
(80K ev/s/core)
at least 5mins 6mins
at most 10mins
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
When to Use Stream
For Fast Analytical & Statistical Jobs The Stream processing layer let you apply a variety of operations
on different event Streams and to compute KPI about users,
services, applications, products or Networks in near RT. The
Stream are processed in real-time according the topologies.
Service ranking
Network failure rate per region
Last week Customer value
AVG Call per region Network monitoring
Real-time BAM
User ranking
Family members or close
friends identification (#Calls
per weeks)
semiocast.com
Example: Knowledge about usage Defining the success of the products by means of patterns
Understanding the usage behaviors
Finding where are the issues
Product Life cycle Evaluation
COUNT &
FILTER
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
When To Use (Huawei) PME
Help operators detect “interesting
situations” in real-time! We can now implement “context-oriented” software which reacts
according to the current contexts. This is the final achievement of
the full service personalization: the service is not only tailored for
you but it is also triggered when you need it or when it is needed!
Intelligent BPM
Intelligent Process Operations
On-line Marketing campaigns
Marketing automation
Dynamic Network Management
Fraud detection
RT traffic management
Google Now like system
Richer IFTTT services
Intelligent alarm system
Intelligent integration (pattern checking) Event-based campaigns
QoE failure detection
REAL
CONTEXT
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
But Still, we need a convergent solution What do I need to implement those cases?
Convergence between Batch Processing & Real-time Processing? More exactly between Batch Data Mining & Real-Time Context. OK, but today what are the architectural patterns developed in the internet world that can achieve that ?
Nothing as such exist!! The Nathan Marz’s Lamda architecture is the closest pattern
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Let’s come back on the Big Data ambiguity All our use cases are Big Data & Intelligent (Machine Learning)
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
The Big Data ambiguity But then we lack of technologies to support our Telco Biz. Cases !
The Twitter lambda architecture is located here ! Let’s see how we could extend it
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
What is the Lambda
Architecture? Description & what is missing for a Telco?
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
What is the Lambda Architecture? Proposed by the Twitter Storm’s architect
Introduction End of 2012 Nathan Marz published the Lambda architecture description used at Twitter for computing Real-time & Hadoop Batch processing. The idea is that the client can query the batch views (e.g. the number of page views for a web site) and merging these results with the real-time views (e.g. the number of page views during the last 10 mins – duration of the batch processing).
Extending Hadoop Stack with Real-Time views
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
What is the Lambda Architecture? Proposed by the Twitter Storm’s architect
Architecture view Integration between Query Focused Data Set (QFD) in batch and in Real-time.
But then No way to implement much more complex Processing ? Such as data mining & RT context?
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
What is the Lambda Architecture? Proposed by the Twitter Storm’s architect
Architecture view Integration between Query Focused Data Set (QFD) in batch and in Real-time.
OK but … This work for counter (SUM, MIN, MAX, etc.) What about data mining & predictive data ?
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Defining the Lambda 2.0 Architecture for Telcos To tackle the business I introduced we need much more than counters
Step 1: Introducing the Context awareness Instead of querying only simple aggregated views on large data, the Pattern Matching Engine enables client to query aggregated views, real-time views and contexts! Now applications can use the batch views within specific situations.
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Defining the Lambda 2.0 Architecture for Telcos Going further the real-time views proposed by the Twitter’s architect
Step 2: Introducing the Data Mining Layer Instead of querying only simple aggregated views on large data, the In-Memory distributed Machine learning enables to compute predicted model & data analytics and exposing predictive models to client, real-time views and contexts! Now applications can use the batch views within specific situations.
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Defining the Lambda 2.0 Architecture for Telcos Example in Real-Time Marketing
Sabri searched for Celine Dion’ s album last week Sabri got 3 M. Jackson’s albums within last month
Sabri searched for 3x M. Carrey’s album last 3 days Sabri is located now in a Music shop
Sabri is going to buy a music album in a shop
These last 2 year Sabri bough 60% of R&B, 15% of Pop and 25% of French Music. The predictive model recommends R&B with a discount between 20 & 25 %
RTD Send an offer by SMS to get R&B music with 20% discount on-line
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Defining the Lambda 2.0 Architecture for Telcos Going further the real-time views proposed by the Twitter’s architect
Step 3: Introducing the Feedback loop 1. Feedback on situation after the model
application 2. Incremental Learning to stick to the
reality
Requires a Machine Learning Back-end that can handle this!
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
High Performance Data Miner SmartMiner
High performance Mining algorithms on 1 Pb Data < 4
mins: is this possible?
High efficiency Increasing prediction accuracy up
to 95%
Incremental learning The predictive model is
continuously processed to fit
the reality
Volume What about applying
algorithms on multiple Pb of
data?
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
What is missing in current Big Data Frameworks?
Map-Reduce-like models are not
so easy for machine learning
key-based operators limits flexibility
Iterations are complex and computationally
expensive
Only batch learning: Scan all data
Cannot handle Online learning algorithms
No Incremental learning by adding
data piece by piece
• High performance Machine Learning platform
› Combine Online and batch Learning
techniques
• Distributed Machine Learning Architecture
› High scalable and distributed paradigm
› Based on very recent research works
› Architecture-aware algorithms
• Algorithm design based on :
› Building sub-models from disjoint subsets of data.
› Parameter Mixing for computing the final model
• Architecture design :
› in-Memory data storage
› Data locality computation
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Huawei Distributed In-Memory Computing
Data Grid
Step 0 : Data
are loaded into
the data grid
Controlle
r
Step 1 :
Launch
learning
signal
Step 1’ : Each Node execute
the training algorithm on data,
locally.
Mixing node
Learner nodes
Step 2 : Each node send,
independently its models
to the mixing node
Step 2’ : The
mixing node
update his model
as the models
arrive
Step 3 :
applying model
to unseen data
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Benchmarking
Type Algorithm Parameters
Dataset
Output Size Rows
Column
s
Clustering K-means/BIRCH
① 20 clusters
② Max iter 20
③ precision 1e-6
47GB 40 million 200 Based on same
precision
Classification Logistic Regression
(w/o feature selection)
① w/o feature
selection;
② Max iter 50
③ Precision 1e-6
47GB 40 million 143 Logistic regression
model
Classification Naive Bayes/RFM N/A 30GB 40 million 100 Classification model
Environment
OS Information SUSE Linux Enterprise Server 11 (x86_64)
OS Core Info linux-244 2.6.32.12-0.7-default
Processor Version Intel(R) Xeon(R) CPU E5-2620
Number of Processors 24
Processor Clock Speed 1.6 GHz
CPU Type 64-bit
Memory Size 400 GB
Hard Disk 2 T
Paging file size / Swap space 2GB
Environment: Huawei RH2288 Server:
Test on single node/multi
nodes with single
thread/multi-thread for
performance and speedup
Algorithm SmartMiner 3 SPARK SmartMiner 2 SM3VS SM2
Speedup
K-means 290s 450s 15300s 53x
Naive Bayes 38s 600s 27180s 715x
Log. Regression 400s 75000s 325500s 814x
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Think about simulating a marketing Campaign on the entire subscriber base…. In few mins…
Why do you need to be 800x Faster? Is it really needed?
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
The double bus pattern The Event bus plays the role of the event collector (as defined in EDA), while the Service Bus abstracts the different services that can be used as actions. The workflow engine guarantees that we can have a workflow of actions when a pattern is recognized. The complex event can also be forwarded back on the Event collector for further consumption.
Integration Strategy How the a PME can be deployed and integrated?
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Huawei Real-Time Context Blueprint architecture
Event
Detection
Event
Process
H. Perf. Data
Mining
RTD Engine
Data
Sources
Online Channels
SMS
USSD
MMS
IVR
Portals
ODPs
Device
Network
Billing/CRM
External
Channels
EDW
SmartCare
Big Data
Hadoop
Exploration
Analyst CS
PS
Probes
Probes
PME Advanced
CEP
Str
ea
m
Fil
ter
Event
Event
Event
Huawei
Stream
Machine
Learning
Prediction
Modeling
Algorithm Suite
Cust. Knowledge
Mgmt.
Big Data Ingestion
Real-time
Batch ETL
In-
memory
& MPP
Content Analysis
Profiling/Tagging
Segmentation
ETL
Provisioning/
Billing
Real-time
Campaign
Design
Planning
Monitor/Eval
Huawei
Channels
UCM
Toolbar
End User
Telco
Conn
Event
Filter
Decision Engine
Automated
Action
Predefine
d Action
Interaction
Execution
Multi-media
Interaction
Platform
Orchestration
ESB
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Examples of Use cases with the PME Illustrations
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
The Premier league Use case Example: The permission-base marketing – the Premier league Use case
Vs
1. Detect the “Fan” fact (location & context) 2. Detect “A fan is moving far from home to see a match” 3. Detect “A goal has been marked by Manchester”
Permission-based Marketing Soccer Example
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
The Premier league Use case Example: The permission-base marketing – the Premier league Use case
Vs
1. Detect the “Fan” fact (location & context) 2. Detect “A fan is moving far from home to see a match” 3. Detect “A goal has been marked by Manchester”
Permission-based Marketing Soccer Example
EVENT Goal(Team: string, MatchLocation :string) PARTITION BY (MatchLocation) EVENT UserAbroad(UserID: string, Location: string, Team: string) PARTITION BY (Location) EVENT HotelRoomOffer(UserID:string, Location :string) PARTITION BY (Location) DATA string FavouriteTeam(UserID: string)="" PARTITION BY (UserID) EMIT FreeBeerOffer(UserID:=$id, Location := $loc) SELECT AT LEAST 1 UserAbroad(@UserID = $id, @Location = $loc, GET isFootBallFan(UserID := @UserID) = true) WITHIN 90 MINUTES AS $out AND AT LEAST 1 Goal(@MatchLocation = $loc, @Team = $team) WITHIN LESS THAN 90 MINUTES AFTER $out[0] CHECK $team = GET FavouriteTeam(UserID:=$id)
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Defining the Mood of users User Segmentation by their current context and usages
The Use Case Assume that we receive web traffic information and we try to segment users based on that traffic. We assume that there has been prior work to determine the kinds of profile we want to recognize. Say we want to capture “female mood”, defined as “a user who visited female-specific websites three times in the last hour and who searched for female terms two times in the last hour”. Independently of the actual sex of the user, we want to tailor the advertisements we show him to his current mood.
EMIT HappyUser ( UserID := $id ) SELECT AT LEAST 3 Visited ( @UserID = $id, true = GET URLHasTag (URL := @URL, Tag := "Happy" ) ) WITHIN LESS THAN 1 HOUR AS $us AND AT LEAST 2 Searched ( @UserID = $id , true = GET WordHasTag (Word := @Word, Tag := "Happy" ) ) WITHIN LESS THAN 1 HOUR AFTER $us [ 0 ]
Big bang Theory
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Product Roll-up Tracking Product usage tracking
Goal One of our Chinese customer wanted to profile a “comic product” in order to define ASAP the best product customization to each consumer. The Business Goal is to customize the product in order to improve customer experience, increase service penetration and increase cross-sales.
A fictional Gaming service In order to show the relevance of the proposition we consider a concrete example of a mobile game resold by the Teclo to Customer. The game is composed by different levels and uses a freemium model. The user can buy artifacts (pets, swords, bike, etc.) and can share it through invitation to friends.
Thanks to “realtime big data”, it is possible to automatically create and apply knowledge about products and consumers in order to maximize the revenue according to the subscriber base behavior and the product catalog.
Real time big data for optimizing service usage & revenue
The Goal is to (1) create and infer knowledge about the games and users, (2) calculate or infer knowledge about game usage, (3) rank the games, (4) Apply this knowledge for a better QoE and increasing sales.
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Product Roll-up Tracking Defining the patterns to recognize
Game Success for gamer From the installation date, the user has played at least 10x within 3 days, shared the game with at least 2 friends and bought 2 artifacts.
Game Middle-Success for gamer From the installation date, the user has played between 2 and 5x within 3 days, did not share the and did not bought any artifacts
Game Middle-Success & leaving From the installation date, the user has played between 2 and 5x within 3 days, did not share the and did not bought any artifacts, and did not connect within 2 days after the last play. Or the user has played more than 10x last 2weeks and has not played within 4 days and he is not abroad.
Game Failure From the installation date, the user has played at most 2x within 3 days, did not share the and did not bought any artifacts.
Cannot be expressed by traditional CEP technologies
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Business cases that we can now realize
without coding Pattern query example
EMIT Success(GameID := $gId)
SELECT AT LEAST 1 Installed(@GameID = $gId, @UserID = $id) WITHIN 3 DAYS AS $installed
AND AT LEAST 10 Opened(@GameID = $gId, @UserID = $id) WITHIN 3 DAYS AFTER $installed[0]
AND AT LEAST 2 BonusBought(@GameID = $gId, @UserID = $id) WITHIN 3 DAYS AFTER $installed[0]
AND AT LEAST 2 InvitationSent(@GameID = $gId, @UserID = $id) WITHIN 3 DAYS AFTER $installed[0]
EMIT Middle(GameID := $gId)
SELECT AT LEAST 1 Installed(@GameID = $gId, @UserID = $id) WITHIN 3 DAYS AS $installed
AND AT LEAST 10 Opened(@GameID = $gId, @UserID = $id) WITHIN 3 DAYS AFTER $installed[0] as $play
AND NO BonusBought(@GameID = $gId, @UserID = $id) WITHIN 2 DAYS AFTER $play[-1]
AND NO InvitationSent(@GameID = $gId, @UserID = $id) WITHIN 2 DAYS AFTER $play[-1]
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Conclusion & Future works (Current R&D) Real-Time Analytics
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Conclusion Real-Time Analytics
New Business cases means new Technological Directions The Pattern Matching Engine and an Incremental High performance ML platform can contribute significantly to this paradigm shift.
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Future Works & R&D Real-Time Analytics
Automated Learning Data PaaS
Boosting Accuracy with Representation learning/ Deep Learning
Anticipation of situations – Probabilistic Predicted Events (Pattern)
Context learning
Consumer behavior learning
Thank you www.huawei.com
http://twiter.com/sskhiri
https://www.linkedin.com/profile/view?id=6710531
https://github.com/sskhiri
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
History of CEP over 10Y Starting from Event Stream to data mining
2003
2005 2006
2012
(1) Stream processing only (2) Pattern matching only as Cont. queries
Pattern & state matching
SmartSwitch PME