ion wed pm

8/3/2019 ion Wed PM

1/90

IBM Haifa Research Lab Event Processing

2010 IBM Corporation

Event processing past, present, future

VLDB 2010 Tutorial,Singapore, September 15th, 2010

Opher Etzion ([email protected])

8/3/2019 ion Wed PM

2/90


2010 IBM Corporation2

An example to kick-off this tutorial:

Act:Passenger

has been re-routed toanotherdestination send theluggage

Act: Bag hasreached tothe wrongaircraft

Notify: Baghas beenchecked butdid not reachthe ULDwithin 20minutes

Notify: Baghas beenchecked butdid not reachtheconnecting

flight

THE LUGGAGE PERSPECTIVE:

Across the 24 largest airlines morethan 5.6 million bags went missing in2006, this is an average of 15.7 bagsper 1,000 travelers. 15% of the

bags are never found.BBC News, April 4, 2007

Event processing canhelp here..

8/3/2019 ion Wed PM

3/90



Another example: premature baby monitoring

Patient is hooked up to multiplemonitors (in hospital or athome) - the physician can setup event- based rules onmultiple measurements and

patients history when to sendan alert and to whom:

DefinedPattern

8/3/2019 ion Wed PM

4/90



Outline of this tutorial

What is behind event processing ? How is it

related to other computing terms? Where are its

roots?

Event processing architecture, building

blocks

The present: State of the practice in event

processing languages, implementation issues,

challenges in implementing event processing

applications

The future: Trends and research challenges

8/3/2019 ion Wed PM

5/90



What is behind event processing ? How is it

related to other computing terms? Where are itsroots?

8/3/2019 ion Wed PM

6/90



What is event processing anyway?

or

Event processing is a form of computing thatperforms operations on events

8/3/2019 ion Wed PM

7/90



In computing we ptocessed events since early days

Network andSystemManagement

8/3/2019 ion Wed PM

8/90



Emerging technologies in enterprise computing(Gartner Hype Cycle, Summer 2009)

8/3/2019 ion Wed PM

9/90



Whats new?The analog: moving fromfiles to DBMS

In recent years architectures, abstractions, and dedicated

commercial products emerge to support functionality that wastraditionally carried out within regular programming.For some applications it is an improvement in TCO; for others isbreaking the cost-effectiveness barrier.

8/3/2019 ion Wed PM

10/90



What is an event

There are various definitions of events

Event is an occurrence within a particular system ordomain; the word event has double meaning: the real-worldOccurrence as well as its computerized representation

8/3/2019 ion Wed PM

11/90



In daily life we often react to events..

8/3/2019 ion Wed PM

12/90



Sometime we even react to the occurrence

of multiple eventsI closed

the deal with the

Canadians

We closed two huge deals in a single day,

It is a good opportunity to send all the team to

Some fun time in Singapore

I closed the deal with the

Australians

8/3/2019 ion Wed PM

13/90



EventPatterns

Pattern detection is one of the notable functions ofevent processing

8/3/2019 ion Wed PM

14/90



What we actually want to react to are situation

TOLL

VILOATOR FRUSTRATEDCUSTOMER

Sometimes the situation isdetermined by detecting thatsome pattern occurred in theFlowing events.

Toll violation

Frustrated customer

Sometimes the events can

approximate or indicate withsome certainty that thesituation has occurred

8/3/2019 ion Wed PM

15/90



Event processing is being used for variousreasons


IBM Haifa Research Lab Event Processing 2008 IBM Corporation

Detect

Decide

Respond

EP Solution Segments Business Value

Real-Time

Operational

Information

Dissemination

Observation

PredictiveProcessingActive Diagnostics

Reactions to events are done as part of business

transactions achieving low latency decisions,

and quick reaction to threats and opportunities

Getting the

right

information

in the rightgranularity

to the right

person at

the right

time

Diagnose

problems

based on

symptoms and

resolve them

Quick observation into exceptional business

behavior and notification to the appropriate

people.

Mitigate oreliminate

predicted events

8/3/2019 ion Wed PM

16/90



Event Driven Architecture

Event driven architecture: asynchronous, decoupled; eachcomponent is autonomic.

8/3/2019 ion Wed PM

17/90



Ancestor: active databases

On event

When condition

Do actionWith coupling mode

Compositeevents wereinherited toeventprocessing

8/3/2019 ion Wed PM

18/90



Ancestor: Data Stream management system

Source: Ankur Jains website

8/3/2019 ion Wed PM

19/90



Ancestor: Temporal databases

There is a substantial temporalnature to event processing.Recently also spatial andspatio-temporal functions arebeing added

8/3/2019 ion Wed PM

20/90



Ancestor: Messaging pub/sub middleware

8/3/2019 ion Wed PM

21/90



Ancestor: Discrete event simulation

8/3/2019 ion Wed PM

22/90



Ancestor: Network and system management

8/3/2019 ion Wed PM

23/90



Event processing architecture, building blocks

8/3/2019 ion Wed PM

24/90

24

Flower Store

Van Driver

Ranking and

Reporting

System

Bid Request

Delivery Bid

Assignments,

Bid alerts, Assign

Alerts

Control System

GPS Location

Location Service

Location

Drivers Guild

Ranking and reports

Delivery

confirmation

Pick Up confirmation

Ranked drivers /

automatic assignment

Bid System

StorePreferences

Delivery Request

Assignment

System

Manual AssignmentAssignment

Assignments,

Pick Up Alert

Delivery Alert

Fast Flower Delivery

8/3/2019 ion Wed PM

25/90



Event processing network

Event

Producer 1Event

Consumer 1

Event

Consumer 2

Event

Producer 2Event

Consumer 3

Agent 2

Channel

Agent 1

State

Agent 3

8/3/2019 ion Wed PM

26/90



Example of EPN part of the FFD example

Automatic

assignment

Driver

Bid routing

Storereference

Manual

assignment

preparationBid Request

channel Bidenrichment

Assignmentmanager

Alerts

channel

Assignment

channel

Assignment

request

channel

Driverstatus

Driver

enrichment

8/3/2019 ion Wed PM

27/90



Event Processing AgentContext

Event

Channel

Event

ConsumerEvent

Type

Event

Producer

Global

State

The seven

Building blocks

8/3/2019 ion Wed PM

28/90



Event type definition

Header Payload Open content

System defined event

attributes

Attributes specific to the

event type

Additional free format

data included in the

event instance

8/3/2019 ion Wed PM

29/90



Producer State Observer in workflows

State observer

Push:

Instrumentation points;

Pull:

Query the state

8/3/2019 ion Wed PM

30/90



Producer Code instrumentation

8/3/2019 ion Wed PM

31/90



Producer syndication

8/3/2019 ion Wed PM

32/90



Producers streams to events

8/3/2019 ion Wed PM

33/90



Producer sensors

8/3/2019 ion Wed PM

34/90



Consumer - Performance monitoring dashboard

8/3/2019 ion Wed PM

35/90



Consumer - Ambient Orb

8/3/2019 ion Wed PM

36/90



Producer and consumer - Sixth sense

8/3/2019 ion Wed PM

37/90



Twitter as a consumer

8/3/2019 ion Wed PM

38/90



Event Processing Agent

Filter TransformDetect Pattern

Translate Aggregate Split Compose

Enrich Project

Event Processing Agents

8/3/2019 ion Wed PM

39/90



The EPA picture

Output

Not selected

Instanceselection

Context

expression

Pattern detect EPA

Relevancefiltering

Input terminalfilter expression

Relevant event types

Derivation Derivationexpression

MatchingPattern signature:Pattern typePattern parameters

Relevant event types

Pattern policies

Patternmatching set

Participantevents

8/3/2019 ion Wed PM

40/90



Filter EPAA filter EPA is an EPA that performs

filtering only, and has no matching or

derivation steps, so it does not

transform the input event.

Filtering

Filter EPA

Filtered Out

Filtered In

Non-Filterable

Principal filter

expression

8/3/2019 ion Wed PM

41/90



Transform EPA sub types

Translate Compose

Aggregate Enrich

Split

Project

8/3/2019 ion Wed PM

42/90



Sample of pattern types

all pattern is satisfied when the relevant

event set contains at least one instance of

each event type in the participant set

any pattern is satisfied if the relevant event

set contains an instance of any of the event

types in the participant set

absence pattern is satisfied when there areno relevant events

relative N highest values pattern is satisfied

by the events which have the N highest value

of a specific attribute over all the relevant

events, where N is an argument

value average pattern is satisfied when the

value of a specific attribute, averaged over all

the relevant events, satisfies the value

average threshold assertion.

always pattern is satisfied when all the

relevant events satisfy the always pattern

assertion

sequence pattern is satisfied when

the relevant event set contains at

least one event instance for each

event type in the participant set, and

the order of the event instances is

identical to the order of the event

types in the participant set.

increasing pattern is satisfied by an

attribute A if for all the relevant

events, e1

8/3/2019 ion Wed PM

43/90



Pattern detection example

Pattern name: Manual Assignment Preparation

Pattern Type: relative N highest

Context: Bid Interval

Relevant event types: Delivery Bid

Pattern parameter: N = 5; value = Ranking

Cardinality: Single deferred

Find the fivehighest bidswithin the

bid interval

Taken fromthe Fast

FlowerDelivery usecase

8/3/2019 ion Wed PM

44/90



Why do we need policies - A simple example:heavy trading scenario

Given:

A stream of events of a singletopic, about the activity in thestock market for a certain stock.

An event is produced every 10minutes when there is trade inthe stock.

Each event consists of: quote(current stock-quote), volume (anaccumulated volume of tradedevents within these 10 minutes).

A selection specification:trigger an automatic tradeprogram if the volume exceeds300,000 3 times within an hour;pass as an argument the last

quote and the sum of the 3volume values.

Event-Id Time-Stamp Quote Volume

E1 9:00 33.23

E2 9:10 33.04 320,000

E3 9:20 33.11 280,000

E4 9:30 33.01 400,000

E5 9:40 32.90 315,000

E6 9:50 33.04 320,000

E7 10:00 33.20 303,000

E8 10:10 33.33 219,000

E9 10:20 33.11 301,000

E10 10:40 33.00 210,000E11 10:50 32.78 400,000

E12 11:00 32.70 176,000

How many times the trade programming is triggered ;

Which arguments are used in each triggering?

Why defining patterns is not thateasy? Because we need to tune upthe semantics

8/3/2019 ion Wed PM

45/90



Pattern policies

Evaluation policyThis determines when the matching sets are produced

Deferred

Cardinality policyThis determines how many matching sets are produced

within a single context partition

Repeated type policyThis determines what happens if the matching stepencounters multiple events of the same type

Every - Override First Last with max/min value of

8/3/2019 ion Wed PM

46/90



Pattern policies cont.

Consumption policyThis specifies what happens to a participant eventafter it has been included in a matching set

Consume Reuse Boundedreuse

Order policyThis specifies how temporal order is defined

By occurrence time by detection time by stream position by attribute

8/3/2019 ion Wed PM

47/90



Our entire culture is context sensitive

In the play The Tea house of the August Moonone of the characters says: Pornographyquestion of geography

This says that in different geographicalcontexts people view things differently

Furthermore, the syntax of the language (noverbs) is typical to the way that the people ofOkinawa are talking

When hearing concert people are not talking,

eating, and keep their mobile phone on silent .

8/3/2019 ion Wed PM

48/90



Context has three distinct roles (which may be combined)

Partition the incoming events

The events that relate toeach customer areprocessed separately

Grouping events together

Different processing forDifferent contextpartitions

Determining the processing

Grouping together eventsthat happened in thesame hour at the samelocation

8/3/2019 ion Wed PM

49/90



Context Definition

A context is a named specification of conditions that groups eventinstances so that they can be processed in a related way. It assignseach event instance to one or more context partitions.

A context may have one or more context dimensions.

A context is a named specification of conditions that groups eventinstances so that they can be processed in a related way. It assignseach event instance to one or more context partitions.

A context may have one or more context dimensions.

TemporalTemporal

SpatialSpatial

State OrientedState Oriented

Segmentation OrientedSegmentation Oriented

8/3/2019 ion Wed PM

50/90



Context Types Examples

Spatial

State Oriented

Temporal

ContextContext

Every day between 08:00

and 10:00 AM

A week after borrowing a disk

A time window bounded by

TradingDayStart andTradingDayEnd events

3 miles from the trafficaccident location

Within an authorized zone ina manufactory

All Children 2-5 years oldAll platinum customers

Airport security level is redWeather is stormy

Segmentation Oriented

8/3/2019 ion Wed PM

51/90



Context Types

Fixed locationFixed location

Entity distance locationEntity distance location

Event distance locationEvent distance location

Spatial

State Oriented

Fixed intervalFixed interval

Event intervalEvent interval

Sliding fixed intervalSliding fixed interval

Sliding event intervalSliding event interval

Temporal

Segmentation Oriented

ContextContext

8/3/2019 ion Wed PM

52/90



The present: State of the practice in event

processing languages, implementation issues,challenges in implementing event processing

applications

IBM H if R h L b E P i

8/3/2019 ion Wed PM

53/90



An ObservationThe Babylon Tower symbolizes the tendencyOf humanity to talk in multiple languages.

The Event Processing area is no different: most languages in the industry really followthe hammer and nails syndrome and extended existing approaches

imperative script language SQL extensions Extension of inference rule language

The epts language analysis workgroup is aimed to understand the various stylesAnd extract common functions that can be used to define what is an eventprocessing language; this tutorial is an interim report

It does not seem that well succeed to settleIn the near future around a single programming style

IBM H if R h L b E t P i

8/3/2019 ion Wed PM

54/90



Existing Styles for EP languages (samples)

InferenceRules

Stateoriented

AgentO

riented

Imperative/Script Based

RuleCore

Oracle

Aleri Streambase

Esper

EventZero

TIBCOWBE

Apama

Spade

AMiT

Netcool Impact

* - if we add simple and mediated event processing the picture is even more diversified

ECARules

SQLextension

Coral8

Agent LogicStarview

XChangeEQProva


8/3/2019 ion Wed PM

55/90



StreamBase Studio


8/3/2019 ion Wed PM

56/90



StreamBase Pattern Matching


8/3/2019 ion Wed PM

57/90



CCL Studio (Coral8 Sybase)


8/3/2019 ion Wed PM

58/90



CCL Pattern Matching

RFID monitoring application

Checks if a tag has been seen by readers A and B, thenC, but not D, within a 10 second window.

Insert into StreamAlertsSelect StreamA.idFrom StreamA a, StreamB b, StreamC c, StreamD dMatching [10 seconds: a && b, c, !d]

On a.id = b.id = c.id = d.id


8/3/2019 ion Wed PM

59/90



Microsoft Streaminsights

var topfive = (from window in inputStream.Snapshot()from e in window

orderby e.f ascending, e.i descending

select e).Take(5);

var avgCount = from v in inputStreamgroup v by v.i % 4 into eachGroup

from window in eachGroup.Snapshot()select new { avgNumber = window.Avg(e => e.number) };


8/3/2019 ion Wed PM

60/90



Esper EPL FFD Example

/** Not delivered up after 10 mins (600 secs) of the request target

delivery time

*/

insert into AlertW(requestId, message, driver, timestamp)

select a.requestId, "not delivered", a.driver, current_timestamp()

from pattern[every a=Assignment (timer:interval(600 + (a.deliveryTime-

current_timestamp)/1000) and

not DeliveryConfirmation(requestId = a.requestId) and

not NoOneToReceiveMSG(requestId = a.requestId))

];


8/3/2019 ion Wed PM

61/90



ruleCore - Reakt

Event stream view - a unique context of events

a view contains a window into the inbound stream of events andcontains commonly only semantically related events

Situation - an interesting combination of multiple events as they occur over time

An item with an RFID tag being picked up from the shelf and then moving past the checkout without being paid for

Rule - an active event processing entity reacting to specific combinations of inbound events over time

Action - the last part of a rule's evaluation in response to a detected situation


8/3/2019 ion Wed PM

62/90



Amit Terminology

Situation

Conditions

Lifespanee55 ee88

Initiator Terminator

Event SelectionEvent Selection ActionsActionsOperationOperation

Keys

ee11

ee33

Input

events

Operator

Joining

Counting

TemporalAbsence

Aggregation

Notifications

Messages

Definition updatesUser plug-ins

ee22


8/3/2019 ion Wed PM

63/90



Amit - Situation


8/3/2019 ion Wed PM

64/90



IBM Websphere Business Events


8/3/2019 ion Wed PM

65/90



Apama EPL FFD Examples


8/3/2019 ion Wed PM

66/90



Apama Simulation Studio cont.


8/3/2019 ion Wed PM

67/90

g


Non functional properties - scalability

scalability in the volume of processed events

Scalability in the quantity of agents Scalability in the quantity of producers

scalability in the quantity of consumers

scalability in the quantity of context partitions

Scalability in context state size

Scalability in the complexity of computation scalability in the processor environment

Scalability is the capability of a system to adapt readily to a greater or lesser intensity of use, volume,

or demand while still meeting its business objectives.


8/3/2019 ion Wed PM

68/90

g


Non functional requirements - security

Ensuring only authorized parties are allowed to be event producersof event consumers Ensuring that incoming events are filtered so that authorized

producers cannot introduce invalid events, or events that they arenot entitled to publish

Ensuring that consumers only receive information to which theyare entitled. In some cases a consumer might be entitled to seesome of the attributes of an event but not others.

Ensuring that unauthorized parties cannot add new EPAs to thesystem, or make modifications to the EPN itself (in systems wheredynamic EPN modification is supported)

Keeping auditable logs of events received and processed, or otheractivities performed by the system.

Ensuring that all databases and data communications links used bythe system are secure.


8/3/2019 ion Wed PM

69/90

g


Performance objectives

Objective

Number

Objective Name Objective metrics

1 MAX input

throughput

Maximize the quantity of input events processed by a

certain system or sub-system within a given time period

2 MAX output

throughput

Maximize the quantity of derived events produced by a

certain system or sub-system within a given time period

3 MIN average

latency

Minimize the average time it takes to process an event

and all its consequences in a certain system or sub-

system

4 Min maximal

latency

Minimize the maximal time it takes to process an event

and all its consequences in a certain system or sub-

system

5 Latency

leveling

Minimize the variance of processing times for a single

event or a collection of events in a certain system or sub-system

6 Real-time

constraints

Minimize the deviation in latency, from a given value,

for the processing of an event and all its consequences in

a certain system or sub-system.


8/3/2019 ion Wed PM

70/90

g


Optimizations

Optimizations related to EPA assignment: partition, parallelism,distribution and load shedding.

Optimization related to the execution process: scheduling, routingoptimizations and load shedding

Optimizations related to the coding of specific EPAs: code optimization,state management.


8/3/2019 ion Wed PM

71/90


Putting derived events in order

Auction Context Interval time

Bidevent

Auction

entry

Anomaly 1

Bid

event

s

Auction

entries

Anomaly 2


8/3/2019 ion Wed PM

72/90


Ordering in a distributed environment -

possible issuesThe occurrence time of an event is accurate, but the event

arrives out-of-order and processing that should haveincluded the event might already been executed.

Neither the occurrence time nor detection time can be trusted,so the order of events cannot be accurately determined.


8/3/2019 ion Wed PM

73/90


Solutions:

Buffering techniques: Assumptions:

Events are reported by the producersas soon as they occur;

The delay in reporting events to thesystem is relatively small, and canbe bounded by a time-out offset;

Events arriving after this time-outcan be ignored.

Principles:

Let X be the time-out offset,according to the assumption it is safeto assume that at any time-point t,all events whose occurrence time isearlier than t - X have alreadyarrived.

Each event whose occurrence timeis To is then kept in the buffer untilTo+X, at which time the buffer canbe sorted by occurrence time, andthen events can be processed in thissorted order.

Retrospectivecompensation:

Find out all EPAs that havealready sent derived eventswhich would have beenaffected by the "out-of-

order" event if it had arrivedat the right time. Retract all the derived

events that should not havebeen emitted in their currentform.

Replay the original events

with the late one inserted inits correct place in thesequence so that the correctderived events aregenerated.


I t t i

8/3/2019 ion Wed PM

74/90


Inexact event processing

uncertainty whether an event actually occurred inexact content in the event payload

inexact matching between derived events and the situations they purport todescribe

Source

malfunction

Malicious

source

Projection of

temporal

anomalies

Imprecise

source

Sampling orestimate

Propagation

of

inexactness

Uncertain

event

Inexact event

content


8/3/2019 ion Wed PM

75/90


False positives and false negatives

False positive situation detection refers to cases in which anevent representing a situation was emitted by an eventprocessing system, but the situation did not occur inreality.

False negative situation detection refers to cases in which asituation occurred in reality, but the event representing thissituation was not emitted by an event processing system


8/3/2019 ion Wed PM

76/90


Retraction

DeliveryRequest

issued

BidRequest

issued

Assignmentissued

Pick-upoccurred

Deliveryoccurred

Order Cancellation

Cancelbid

preparation

Cancelbidandassignmentprocess

Notifyassigneddriverabout

cancellation

Und

oable

Maystillbeca

ncelledwithsome

penalty


8/3/2019 ion Wed PM

77/90


The future: Trends and research challenges


8/3/2019 ion Wed PM

78/90


Trend I: Going from narrow to wide

Border security radiation detection

Mobile asset geofence

Logistic and scheduling

Unauthorized use of heavymachinery

Hospital patient and asset tracking

Activity monitoring for taxing andfraud detection

Intelligent CRM in banking

EDA and asynchronous BPM inretail

Situation awareness in energy

utilities

Situation awareness in airlines

Reduce cost in injection therapy

Next generation navigation

Real-time management ofhazardous materials

Finding anomalies in point of salesin retail stores

Elderly behavior monitoring

Some recentlyreported applications

(EPTS use-cases WG)

Source: ebizQ Event processing market pulse


8/3/2019 ion Wed PM

79/90


Trend I: Going from narrow to wide

Taking event processing outside enterprise computing:

Robotics

Bio-Informatics

Socio-technical systems

HomeAutomation


8/3/2019 ion Wed PM

80/90


Trend II: Going from monolithic to

diversified

Variety of functions

Variety of Quality of Servicerequirements:

Variety of platforms

One size fits all will not work Instead a collection of building blocksthat can fit together


8/3/2019 ion Wed PM

81/90


Trend III: Going from proprietary tostandard-based standard directions

The current situation: Babylon tower: variety oflanguages, event representation

Serves as enabler to achieving other trends and

general maturity

The shift of vendors from start-up dominant tobigger companies makes the atmosphere more

friendly towards standards.

Areas for

Standards:ModelingEvent representationInteroperabilityLanguages PIM


T d IV G i f t d t

8/3/2019 ion Wed PM

82/90


Trend IV: Going from programmer centered tosemi-technical person centered

Source: ebizQ Event processing market pulse


8/3/2019 ion Wed PM

83/90


Trend V: Going from stand-alone toembedded

Packaged applications

Business ActivityMonitoring

Sensor Platform

Middleware and platforms


8/3/2019 ion Wed PM

84/90


Trend VI: Going from reactive to proactive

TRAFFIC JAM

End of game


8/3/2019 ion Wed PM

85/90


Emerging directions: Four directions to observe

Multiple platforms samelook and feel

The engineering ofconstructing EPapplications

Adding intelligence toEvent processing

Tailor-made optimizations


8/3/2019 ion Wed PM

86/90


Emerging direction I: multiple platforms samelook and feel

ApplianceApplianceStream

Platform

Stream

PlatformCloud

computing

Platform

Cloudcomputing

Platform

ESB /

Messaging

Platform

ESB /Messaging

Platform

EmbeddedEmbedded

Virtual Event Processing Platform


8/3/2019 ion Wed PM

87/90


Emerging direction II: Tailor-madeoptimizations

Local optimizations: each EPA will be optimized for its ownpurpose / assumptions / QoS indicators average/worstcase latency, input/output throughput

Global optimization: scheduling, load balancing,assignment

EPA EPA

EPA

EPA

Producer

Producer

EPA

Consumer

Consumer

Local optimizations

Global optimizations


8/3/2019 ion Wed PM

88/90


Emerging direction III:Event processing software engineering

Best practices

MethodologiesModeling

& meta-modeling

DesignPatterns


8/3/2019 ion Wed PM

89/90


Emerging direction IV:Intelligent event processing

Offline andcontinuous miningof meaningful

patterns in eventhistories

Inexact eventprocessing handling inexactevents and alsofalse positives andfalse negatives

Causality a key

for proactive, butalso vital forprovenance


8/3/2019 ion Wed PM

90/90

Summary

Event processinghas emerged fromsome academic

disciplines

The state of thePractice is the 1st

generation ofproducts mainly

engineering based

Going to the nextphase manychallenges thatrequire collaboration

of research andindustry

Already attracted coverage ofanalysts and all major softwarevendors

ion wed pm

Documents