epts debs2012 event processing reference architecture - design patterns v2_04b
DESCRIPTION
Presentation of functional design patterns for event processing with examples from TIBCO, Oracle, IBM and Prova event processing technologies. Superset of the Tutorial at DEBS2012. Editors Alex Alves, Catherine MoxeyTRANSCRIPT
epts event processing technical society
epts event processing technical society
Adrian Paschke (Freie Universitaet Berlin) Paul Vincent (TIBCO Software) Catherine Moxey, Martin Hirzel, Dave Rushall, Richard Jacks (IBM) Alex Alves (Oracle)
Event Processing
Reference Architecture
- Design Patterns
Patterns v2.04b
Content by: members of the
EPTS Reference Architecture Working Group
epts event processing technical society
2 2
Agenda
• EPTS Reference Architecture Introduction
• EP Patterns:
– EPTS RA patterns
– Some Use Cases
– EPIA patterns
epts event processing technical society
3
Event Processing, Complex Event Processing, Event Stream Processing
Event Abstraction, Event Pattern
Detection, Event Composition
etc. etc.
Introduction to Event Processing
Event
Managem
ent
Event Consumer
(Event sink, event handler, event
listener)
Run time Administration
Event
Type D
efinitio
ns,
Event
Pro
cessin
g R
ule
s,
etc
.
Design time
Events Derived
Events
Event Producer
(Event Source, Event Emitter)
epts event processing technical society
4
Reference Architecture: Functional View
Event Production Publication,
Retrieval
Even
t Pro
cess M
on
itorin
g, C
on
trol
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
Complex Event Detection Consolidation, Composition,
Aggregation
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
Event Consumption Dashboard, Apps, External Reaction
Run time Administration
Even
t a
nd
Co
mp
lex E
ven
t (P
att
ern
, Contr
ol, R
ule
, Q
uery
, RegEx.e
tc)
D
efi
nit
ion
, M
od
eli
ng
, (co
nti
nu
ou
s) I
mp
ro
vem
en
t
Design time
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
0..*
0..*
0..*
0..*
Sta
te M
an
ag
em
en
t
epts event processing technical society
5
Comparison of EPTS RA Fns and EPIA EP Agents
5
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
Complex Event Detection Consolidation, Composition,
Aggregation
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
EPA
Filter Transformation
Translate
Enrich Project
Aggregate Split Compose
Pattern detect
epts event processing technical society
6 6
Agenda
• EP Meta Pattern / Pattern Structure
epts event processing technical society
7
What do we want to cover in an EP Pattern?
7
Event Filtering: a stream or list of events may be filtered
on some payload or metadata information
such that some subset is selected for further processing.
Type /
synonym
Operation /
Operators
Operands
Rationale /
business rule Mapping to
EPL template
epts event processing technical society
8
EP MetaPattern / Pattern Description (version 1)
• Name & Alternative Names
• Classification versus other references
– EPTS RA and other references e.g. “EP in Action” book
• Description
– Role, and Business Rule type specification / requirement
• Structure
– EPTS Glossary terms
• Implementations
8
epts event processing technical society
9
EP MetaPattern future versions?
• Name & Alternative Names
• Classification versus other references
– EPTS RA and other references e.g. “EP in Action” book
• Description
– Role, and Business Rule type specification / requirement
• Structure
– EPTS Glossary terms
• Implementations
9
Extend examples
pseudocode – real example
may revise/improve EP Fns
classification / ontology
compare/extend
other “event patterns”
epts event processing technical society
10
EP Pattern Implementations
• Differ in Implementation type (Event Processing
Language, Execution semantics, etc)
• Sampled systems we cover:
1.TIBCO BusinessEvents Rete-production-rule-engine
(with continuous query and event-expression options)
2.Oracle Event Processing Stream-processing engine
3.Prova Logic Programming Semantic CEP Rule Engine
4.IBM Decision Server Events component of WebSphere
Operational Decision Management
& IBM InfoSphere Streams stream processing engine
10
epts event processing technical society
Implementations:Sampled Systems: TIBCO BusinessEvents
• TIBCO BusinessEvents “event server” / event processing engines
• Multiple languages / engines in various packaging options
– UML-based event and concept (object) class hierarchical models
– Java-based language
– Eclipse-based IDE
• Distributed execution model:
– Inference agents, Query agents, Datagrid agents
– In-memory, memory+grid, explicit grid operational modes
– “Grid Options” of TIBCO ActiveSpaces / Oracle Coherence
– eXtreme Event Processing use cases: Fedex, US Govt, …
• Transactional model option:
– TIBCO AS Transactions
– eXtreme Event Processing use cases: Telco
• Extensible: eg OWL import, RDF import, etc
epts event processing technical society
Implementations:Sampled Systems: TIBCO BusinessEvents (ctd)
Event Processing
Events
Event Storage
Information Event and Data Structures
States and Transitions
Inference Rules
Sets and Queries Stored
Events and Data
via JMS, RV, MQ, TCP/IP, etc…
Continuously process events using procedural and declarative event processing elements
Temporal Patterns
Decision Tables
epts event processing technical society
13
Implementations:Sampled Systems: Oracle Event Processing (formerly Oracle CEP)
• Development platform for event processing applications
• Application model based on EPN (event processing network) abstraction running on top of OSGi-based Java container.
• Language is an extension of SQL with stream and pattern matching extensions
epts event processing technical society
14
Implementations:Sampled Systems: Oracle Event Processing (formerly Oracle CEP)
• CQL: Continuous Query Language
• Leverages SQL, extended with Stream, Pattern-matching, and Java.
• Continuous: driven by time, and events
• Highly-sophisticated push-down technology to RDBMS, Cache, Hadoop
epts event processing technical society
15
Implementations:Sampled Systems: Prova (http://prova.ws)
• Java JVM based, open source rule
language for reactive agents and event
processing
• Leverages declarative ISO Prolog standard
extended with (event,message) reaction
logic, type systems (Java, Ontologies),
query built-ins, dynamic Java integration.
• Combines declarative, imperative (object-
oriented) and functional programming
styles
• Designed to work in distributed Enterprise
Service Bus and OSGi environments
• Supports strong, loose and decoupled
interaction
• Compatible with rule interchange standards
such as Reaction RuleML
Event Cloud
(unordered events)
new auto pay account login
account login deposit
withdrawal
logout
account balance
transfer
deposit new auto pay
enquiry enquiry
logout
new auto pay account login
account login deposit
activity history
withdrawal
logout transfer deposit new auto pay
enquiry
enquiry
book
request incident
Event Streams (ordered events)
Agent / Service
Interface
Rule Service
Consumer
Message Interchange
0.0000
Request
0.0000
Response
Prova Agent
Prova Agent
…
Computer
Prova
direct API
implementation
Prova Agent
Prova Agent
Prova Agent Prova
Agent
Strong Coupling
Loose Coupling
Decoupled Prova Event
Processing
Network
epts event processing technical society
16
Implementations:Sampled Systems: IBM WODM Decision Server Events • Decision Server Events component of IBM WebSphere Operational
Decision Management (WODM)
• Manages business events flowing across systems and people to provide timely insight and responses
• Detect, evaluate, and respond to events
• Discover event patterns and initiate actions
epts event processing technical society
17
Implementations:Sampled Systems: IBM CICS Transaction Server for z/OS
Event Production Publication, Retrieva,
Identification, Selection, Filtering,
Enrichment, Transforms, Routing
Event Preparation Identification, Selection, Filtering, Monitoring,
Enrichment
Complex Event Detection Consolidation, Composition,
Aggregation
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
Event Consumption Dashboard, Apps, External Reaction
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
CICS TS
epts event processing technical society
18
Implementations:Sampled Systems: IBM InfoSphere Streams – Stream Processing Language
• IBM InfoSphere Streams: platform for analyzing big data in motion i.e. high-volume, continuous data streamsSPL
• IBM Streams Processing Language (SPL)
– Programming language for InfoSphere Streams
– Exposes a simple graph-of-operators view
– Provides a powerful code-generation interface to C++ and Java
Streams Processing
Language and IDE
Runtime
Environment
Tools and
Technology
Integration
Highly Scalable
stream processing runtime
Streams Console & Monitoring,
Built-in Stream Relational Analytics,
Adapters, Toolkits
Streams Studio
Eclipse IDE for SPL
Front Office 3.0
epts event processing technical society
19 19
Agenda
• Functional Reference Architecture-based
EP Patterns
epts event processing technical society
20
Patterns Coverage
Event Production Application Time
Publication, Retrieval
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
Complex Event Detection Consolidation, Composition,
Aggregation
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
Event Consumption Dashboard, Apps, External Reaction
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
0..*
0..*
0..*
0..*
Sta
te M
an
ag
em
en
t
epts event processing technical society
21 21
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
Event preparation is the process of preparing the event
and associated payload and metadata for further stages
of event processing.
For example, event preparation may involve the separation
and discarding of unused event payload data,
and the reformatting of that payload data
for downstream event processing.
EP Patterns:RA:Preparation
epts event processing technical society
22
RA:Preparation:Identification:
Identification: incoming events identified relative to prior
events and event types
– E.g. associating a SMS event with particular mobile phone
account
– E.g. recognizing an event from data (event entity recognition,
extraction) and identifying an event to be of a particular event
type
22
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
23
Preparation:Identification:Classification
• Alternative Names:
– Event Lookup, Event Entity Recognition, Event Extraction
• EPTS Reference Architecture:
– “Incoming events will need to be identified relative to prior
events and event types, such as associating events with
particular sources or sensors or recognizing / extracting
events from data relative to event type information (event
type systems, event ontologies)"
• EPIA:
– Transform, Translate, Enrich / Project
23
Event Preparation Identification, Selection,
Filtering, Monitoring, Enrichment
epts event processing technical society
24
Preparation:Identification:Description
• Role / Explanation
– Associating an event with some existing entity (data or past
event, event type)
• Associated Business Rule Specification
– a selection (to be enforced during the processing of events):
All <event entities> whose <attribute1> matches the
<attribute2> of <some other event or data or type>
is related by <relationship> to that <some other event or
data or type>.
24 event
Identification
f(event)
known
event
Other data
or event or type
Event Preparation Identification, Selection,
Filtering, Monitoring, Enrichment
epts event processing technical society
25
Preparation:Identification:Structure
• EPTS Glossary comparison
– Event type (event class, event definition, or event
schema)
• A class of event objects.
• Event types should be defined within some type definition
system ... will usually specify certain predefined data
(attributes), examples of which might be:
– A unique event identifier used to reference the event
25
Event Preparation Identification, Selection,
Filtering, Monitoring, Enrichment
epts event processing technical society
26
Preparation:Identification:Implementations
• TIBCO, Prova, Oracle and IBM implementations for
this pattern
26
event Identification
f(event)
known
event
Other data
or event
Event Preparation Identification, Selection,
Filtering, Monitoring, Enrichment
epts event processing technical society
27
Preparation:Identification:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event, data
<if> event-data-match
<then> create-event-
association-with-data
• Effect: find appropriate data that
matches the incoming event*
and create a reference between
the event and the data
27
event
Identify
association
Event
Add
association
to event
Entity
community
* note: event object = concept: input events are immutable**
** note: except in PreProcessor
Event Preparation Identification, Selection,
Filtering, Monitoring, Enrichment
epts event processing technical society
28
Preparation:Identification:Implementations: Oracle EP: queries
• General pattern:
SELECT <property>
FROM <SOURCE>
WHERE <predicate-condition>
• Effect: matches events coming from
<SOURCE> with pre-defined event-type
28
event
SOURCE
event
Event-type
Event Preparation Identification, Selection,
Filtering, Monitoring, Enrichment
epts event processing technical society
29
Preparation:Identification:Implementations:Prova Semantic Identification
• General pattern: <rcvMsg / rcvMult> Event ^^ Semantic Type :-
• Effect: find appropriate event entity type that matches the
incoming event data and create a reference between the event
and the event type (from the semantic knowledge), so that the
syntactic (event) data becomes a (semantic) event, which has
relations with the semantic background knowledge base 29
Event Preparation Identification, Selection,
Filtering, Monitoring, Enrichment
Event
Data
Semantic
Identification
f(event)
Semantic
event
Semantic
Knowledge Base
epts event processing technical society
30
Preparation:Semantic Identification:Prova Semantic Rules
Event Stream {(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) } {(Name, “SAP”)(Price, 65)(Volume, 1000) (Time, 2)}
{(OPEL, is_a, car_manufacturer), (car_manufacturer, build, Cars),
(Cars, are_build_from, Metall), (OPEL, hat_production_facilities_in, Germany),
(Germany, is_in, Europe) (OPEL, is_a, Major_corporation),
(Major_corporation, have, over_10,000_employees)}
Semantic Knowledge Base
rcvMult(SID,stream,“S&P500“, inform,
tick(Name^^car:Major_corporation,P^^currency:Dollar,
T^^time:Timepoint)) :- …<further processing of semantic event>.
Event Preparation Identification, Selection,
Filtering, Monitoring, Enrichment
Identify event and associate with
semantic background knowledge
epts event processing technical society
31
Preparation:Identification:Implementations: IBM WODM Decision Server Events
31
Event Preparation Identification, Selection,
Filtering, Monitoring, Enrichment
• Example: event rule relates to ‘new purchase order’ events,
and the events are processed in the context of the customer
number
• Effect: presence of ‘new purchase order’ in the rule
identifies that event type, and the events are processed in
the context of the customer number field – the events of
interest could be said to be identified by ‘new purchase
order’ type and customer number, or the latter could be
regarded as selection
epts event processing technical society
Preparation:Identification:Implementations: IBM InfoSphere Streams (SPL)
32
Event Preparation Identification, Selection,
Filtering, Monitoring, Enrichment
type Kind = enum { knownKind1, knownKind2, knownKind3 }; /*...*/ stream<rstring key, int32 payload> Events = PrepId_EventsSource() {} stream<rstring key, Kind kind> Mappings = PrepId_MappingsSource() {} stream<Kind kind, int32 payload> KnownEvents = Custom(Events; Mappings) { logic state: mutable map<rstring, Kind> _keyToKind = {}; onTuple Events: if (key in _keyToKind) submit({ kind=_keyToKind[key], payload=payload }, KnownEvents); onTuple Mappings: _keyToKind[key] = kind; }
Custom function
Known Events
Filter to map Event
key to known kind Identified
known events
• Example:
stream Events
stream Mapping
epts event processing technical society
33
RA:Preparation:Selection:
Selection: particular events selected for further analysis or
pattern matching
– E.g. Selecting the n’th event as a part of a sampling function
– E.g. Selecting events related to some existing event to allow
enrichment of that existing event
33
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
34
Preparation:Selection:Classification
• Alternative Names:
– Event Query
– See also Event Filtering and Event Identification
• EPTS Reference Architecture:
– “During event preparation, particular events may be
selected for further analysis. Different parts of event
processing may require different selections of events.
See also event filtering. "
• EPIA:
– Filter
34
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
35
Preparation:Selection:Description
• Role / Explanation
– Selecting or locating an event due to some criteria or rules
• Associated Business Rule Specification
– a selection (to be enforced during the processing of events):
All <event entities> whose <attribute1> matches the
<attribute2> of <some other event or data>
is selected for <processing>.
35 event
Selection
f(event)
selected
event
Other data
or event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
36
Preparation:Selection:Structure
• EPTS Glossary comparison
– Event type (event class, event definition, or event
schema)
• A class of event objects.
• Event types should be defined within some type definition
system ... will usually specify certain predefined data
(attributes), examples of which might be:
– A unique event identifier used to reference the event
36
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
37
Preparation:Selection:Implementations
• TIBCO, Prova, Oracle and IBM implementations for
this pattern
37
event Selection
f(event)
selected
event
Other data
or event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
38
Preparation:Selection:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event, data
<if> event-data-match
<then> process-event
• Effect: filter event per selection criteria
(Note: action may be to
mark event as selected)
38
event
Compare event
vs selection
criteria
Event
Entity
community
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
39
Preparation:Selection:Implementations:Prova
• General pattern: <rcvMsg / rcvMult>
(CID,Protocol, Agent,Performativ,Event|Context)
:- <selection_processing> Event.
• Effect: select event according to message context (conversation
id, protocol, agent, pragmatic performativ, event context) (see
also monitoring) and further process selected events, e.g.
comparing them against existing events, data, guards, rules,
selection functions 39
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
Event
Selection
f(message(event))
Selected
Event
Existing Event, Data,
Guards, Rules
Event
Compare Event vs.
selection criteria
((Events, Data, Guards)
epts event processing technical society
40
Preparation:Selection:Implementations:Prova Messaging Reaction Rules in Prova
• Send a message
sendMsg(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive a message
rcvMsg(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive multiple messages
rcvMult(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)
Selection according to:
– CID is the conversation identifier
– Protocol: protocol e.g. self, jms, esb etc.
– Agent: denotes the target or sender of the message
– Performative: pragmatic context, e.g. FIPA ACL
– [Predicate|Args]|Context or Predicate(Arg1,..,Argn): Event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
41
Preparation:Selection:Implementations:Prova Example % Select stock ticker events from stream "S&P500"
% Each received event starts a new subconversation (CID)
which further processes the selected event (select)
rcvMult(CID,stream,“S&P500“, inform, tick(S,P,T)) :-
select(CID,tick(S,P,T)).
% Indefinitely (count=-1) receive further ticker events from
other streams that follow the previous selected event in
event processing group (group=g1). If the price differs
for the same stock at the same time [T1=T2, P1!=P2] then …
select(CID,tick(S,P1,T1)) :-
@group(g1) @count(-1)
rcvMsg(CID,stream, StreamID ,inform, tick(S,P2,T2))
[T1=T2, P1!=P2] %guard: if at same time but different price,
println (["Suspicious:",StreamID, tick(S,P2,T2)]," ").
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
42
Preparation:Selection:Implementations: Oracle EP: queries
• General pattern:
SELECT <property>
FROM <SOURCE>
WHERE <predicate-condition>
• Effect: select certain properties of events
from <SOURCE>
42
event
select
event
Event
properties
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
43 43
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
Preparation:Selection:Implementations: IBM WODM Decision Server Events
• Uses ‘business objects’ to select properties
of incoming events that will be used when
events are processed
• Business Objects allow for a level of
indirection between incoming events and
specifications made by the business about
how to respond to events
• Mapped via the Event Designer tool
epts event processing technical society
44 44
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
type Kind = enum { knownKind1, knownKind2, knownKind3 };
/*...*/
stream<int32 payload> Selected = Functor(KnownEvents) {
param filter : kind == knownKind1;
}
stream<int32 payload> ProcessedEvents = PrepSel_Process(Selected) {}
Preparation:Selection:Implementations: IBM InfoSphere Streams (SPL)
event stream Functor selected
events
Known
Events
Select events
which match
known kind
Kind 1 of
known events
• Example:
epts event processing technical society
45
RA:Preparation:Filter
Filter: filter out all events that have some property in their
payload / data
– E.g. customer purchases: filter out those with values < $100
45
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
46
Preparation:Filter:Classification
• EPTS Reference Architecture:
– "During event preparation, a stream or list of events may
be filtered on some payload or metadata information
such that some subset is selected for further processing."
• EPIA:
– Filter (EPA or Event Processing Agent) performs filtering
only and has no matching or derivation steps, so it does
not transform the input event.
46
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
47
Preparation:Filter:Description
• Role / Explanation
– Comparing some property of the event (an attribute or
metadata) with some other value (from some other event, or
data)
• Associated Business Rule Specification
– As a constraint: a selection (to be enforced during the
processing of events):
All <event entities>
that have
<filter expression>
must have
<some status>. 47
event Filter
f(event) event
discarded
event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
48
Preparation:Filter:Structure
• EPTS Glossary comparison
– A filter pattern can relate to several terms in the EPTS
Glossary
• can be specified in an event pattern (A template containing
event templates, relational operators and variables...)
• .. by an event pattern constraint (A Boolean condition that
must be satisfied by the events observed in a system...)
• .. as part of an event processing rule (A prescribed method
for processing events.)
• .. or as part of event stream processing (Computing on inputs
that are event streams.)
–
48
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
49
Preparation:Filter:Implementations
• TIBCO, Oracle, Prova and IBM implementations for
this pattern
49
event Filter
f(event) event
discarded
event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
50
Preparation:Filter:Implementations: TIBCO BusinessEvents: rules
• General pattern: <declare> event
<if> event-boolean-expression
<then> update-event-with-fact
OR
action-for-filtered-event
• Effect: ignore filtered events
on operations on non-filtered events
50
event
Filter
f(event)
event
Operation on
event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
51
Preparation:Filter:Implementations: TIBCO BusinessEvents: event removal rules
• Implementation type: Rete-based rule-driven system
• Alternate pattern: <declare> event
<if>
NOT event-boolean-expression
<then>
consume-event
• Effect: removed filtered events from
system (and ALL subsequent operations!)
51
event
NOT Filter
!f(event)
event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
52
Preparation:Filter:Implementations: TIBCO BusinessEvents: State transitions
• General pattern: <state models for concept C>
<route from State n to n+1>
<on> event
<where> event-filter
• Effect: change state for some concept
• Notes: this is a specialisation of the
general rule pattern!
52
event
Filter
query(event)
State
transition
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
53
Preparation:Filter:Implementations: TIBCO BusinessEvents: Queries
• General pattern: <select> output
<from> event
<where> event-filter
<policy> notification
• Effect: collect filtered events for
event operations per some policy
• Note: the pattern matching framework
has a similar notion
but a different surface syntax 53
event
Filter
query(event)
Event
subset
Operations on
event subset
on policy event
Policy
event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
54
Preparation:Filter:Implementations: Oracle EP: queries
• General pattern:
SELECT <property>
FROM <SOURCE>
WHERE <predicate-condition>
• Effect: discard events whose predicate
evaluate to false
54
event
predicate
event
Operation on
event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
55
Preparation:Filter:Implementations: Oracle EP: model
55
• Scenario:
– Select only stocks from the stock tick stream whose
symbols are ‘AAA’.
• Application Model
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
56
Preparation:Filter:Implementations: Oracle EP: query
56
Specify STREAM source
Define predicate for
filtering
SELECT *
FROM StockTickStream
WHERE symbol = ‘AAA’
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
57
Preparation:Filter:Implementations: Prova: rules
• Implementation type: backward reasoning logical filter
• General pattern template: <rcvMsg / rcvMult> Event :-
logical filter condition(s),
... .
logical derivation rule 1 :-
filter conditions.
logical derivation rule 2 :-
filter conditions.
...
• Effect: Unify received events with
event pattern definition and apply logical
filter rules (derivation rules with rule chaining) 57
event
Event pattern +
logical rule filter
event
Operation on
event
event
Backward
Resoning
Rule
chaning
…
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
58
Preparation:Filter:Implementations: Prova: Messaging Reaction Rules in Prova
• Send a message
sendMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive a message
rcvMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive multiple messages
rcvMult(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
Description:
– XID is the conversation identifier
– Protocol: protocol e.g. self, jms, esb etc.
– Agent: denotes the target or sender of the message
– Performative: pragmatic context, e.g. FIPA Agent Communication
– [Predicate|Args] or Predicate(Arg1,..,Argn): Message payload
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
59
Preparation:Filter:Implementations: Prova: Example
% Filter for stocks starting with „A“ and price > 100
rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-
S = "A.*",
P > 100,
sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).
% Filter for stocks starting with „A“ and price > 100
rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-
filter(S,P),
sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).
filter(Symbol,Price) :-
Price > 100,
Symbol = "A.*".
Example with rule chaining
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
60
Preparation:Filter:Implementations: Prova: Semantic Event Processing Example
Event Stream {(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) } {(Name, “SAP”)(Price, 65)(Volume, 1000) (Time, 2)}
Semantic Query Filer: Stocks of companies, which have production facilities in Europe and produce products out of metal and Have more than 10,000 employees.
{(OPEL, is_a, car_manufacturer), (car_manufacturer, build, Cars),
(Cars, are_build_from, Metall), (OPEL, hat_production_facilities_in, Germany),
(Germany, is_in, Europe) (OPEL, is_a, Major_corporation),
(Major_corporation, have, over_10,000_employees)}
Semantic Knowledge Base
rcvMult(SID,stream,“S&P500“, inform,
tick(Name^^car:Major_corporation,P^^currency:Dollar,
T^^time:Timepoint)) :- … <semantic filter inference> .
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
61
Preparation:Filter:Implementations: IBM WODM Decision Server Events
61
event
Filter
event
Action
event
• Example:
Define a filter to only process events with a
purchase value greater than £100
• Effect: Only process purchase events
which match the filter, taking the action to
send a discount voucher
event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
62
Preparation:Filter:Implementations: IBM CICS Transaction Server for z/OS application event capture (1)
62
event
Filter
event
Operation on
event
• Example:
Define a filter to only produce events for banking transactions with
a value greater than $5000
• Effect: Only produce events which match the filtering predicates
event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
63
Preparation:Filter:Implementations: IBM CICS Transaction Server for z/OS application event capture (2) Details
63
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
<eventCaptureSpecification>
<name>LargeCashDepositCS</name>
<eventIdentifier>LargeCashDeposit</eventIdentifier>
<description>Capture details for event when large cash deposit is
made</description>
<filter>
<contextFilter>
. . .
</contextFilter>
<locationFilter filterType="PROGRAM_INIT">
<programInit>
<PROGRAM filterValue="CPPEPM22" filterOperator="EQ"
keyword="PROGRAM"/>
<CHANNEL filterValue="" filterOperator="OFF" keyword="CHANNEL"/>
</programInit>
</locationFilter>
<dataFilter>
<filterItem>
<dataFilter filterValue="5000" filterOperator="GT"
languageStructureFilename="" languageStructureName="" languageVariableName=""
dataPrecision="0" dataType="PACKED" length="8" offset="5" container="APP"PX22"
source="CHANNEL"/>
</filterItem>
</dataFilter>
</filter>
<dataCapture>
. . .
</dataCapture>
</eventCaptureSpecification>
epts event processing technical society
64 64
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
stream<rstring name, uint32 age> People = PrepFil_Source() {}
stream<rstring name, uint32 age> YoungPeople = Filter(People) {
param filter : age < 30u;
}
Preparation:Filter:Implementations: IBM InfoSphere Streams (SPL)
event stream Filter event
stream
People Filter age < 30
Young People
• Example:
event stream
Un-Young People
epts event processing technical society
65
RA:Preparation:Monitoring:
Monitoring: particular event channels are monitored to identify
events of interest
– E.g. Selecting a particular channel and event selector in
some middleware subscription
65
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
66
Preparation:Monitoring:Classification
• Alternative Names:
– Event Subscribe
– See also Event Selection
• EPTS Reference Architecture:
– “During event preparation, particular types of events may
be monitored for selection for further processing. This
may utilise specific mechanisms external to the event
processing such as exploiting event production features."
• EPIA:
– Filter
66
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
67
Preparation:Monitoring:Description
• Role / Explanation
– Directing an external agent to select events
– Applying some monitor function over some event channel
• Associated Business Rule Specification
– a monitor (of an event source):
All <event entities> whose <attribute1> matches the
<attribute2> of <some monitoring criteria>
is selected for <processing>.
67 event
Monitor
f(event)
monitored
event
Monitor
criteria
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
68
Preparation:Monitoring:Structure
• EPTS Glossary comparison
– (Example mechanism): Publish-and-subscribe (pub-sub)
• A method of communication in which messages are
delivered according to subscriptions .
• Note 1. Subscriptions define which messages should flow to
which consumers.
• Note 2. Event processing applications may use publish-and
- subscribe communication for delivering events. However,
publish - and - subscribe is not definitional to event
processing – other communication styles may be used.
– Subscriber
• An agent that submits a subscription for publish- and –
subscribe communication.
68
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
69
Preparation:Monitoring:Implementations
• TIBCO, Prova, Oracle and IBM implementations for
this pattern
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
event Monitor
f(event)
monitored
event
Monitor
criteria
epts event processing technical society
70
Preparation:Monitoring:Implementations: TIBCO BusinessEvents: Event Destination
• General pattern: <define> channel,
destination, event
• Effect: subscribe to events on this
channel per this destination
(Note: part of Event definition)
70
Channel
Destination
Event
Entity
channel
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
71
Preparation:Monitoring:Implementations: Oracle EP: Event Destination
• General pattern: <channel>
<name>dest1</name>
<selector>query1</selector>
</channel>
• Effect: subscribe to events on this
channel per selected queries.
71
Channel
Selector
Event
Entity
channel
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
72
Preparation:Monitoring:Implementations: Prova: Selection
• General pattern: <rcvMsg / rcvMult>
(CID,Protocol, Agent,Performativ,Event|Context)
:- <selection_processing> Event .
• Effect: monitor message channels and select (see also
Selection Pattern) events according to message context
(conversation id, protocol, agent, pragmatic performativ, event
context); 72
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
Event
Monitor + Selection
f(message(event))
Selected
Event
Event
epts event processing technical society
73
Preparation:Monitoring:Implementations: Prova: Messaging Reaction Rules in Prova
• Send a message
sendMsg(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive a message
rcvMsg(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive multiple messages
rcvMult(CID,Protocol,Agent,Performative,[Predicate|Args]|Context)
Selection according to:
– CID is the conversation identifier
– Protocol: protocol e.g. self, jms, esb etc.
– Agent: denotes the target or sender of the message
– Performative: pragmatic context, e.g. FIPA ACL
– [Predicate|Args]|Context or Predicate(Arg1,..,Argn): Event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
74 74
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
Preparation:Monitoring:Implementations: IBM WODM Decision Server Events
• Example: Use an event
connector that
subscribes for
aircraft arrivals
from a JMS topic
• Effect: A message
selector is used to
limit which messages
will trigger events,
and an XSL
stylesheet is used to
transform the
message text into an
event packet.
epts event processing technical society
75 75
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
type BargainType = rstring symbol, float32 price;
/* in exporting application */
() as ExportTech = Export(TechSectorBargains) {
param properties : { kind = "bargains", category = "tech" };
}
/* in importing application */
stream<BargainType> TechBargains = Import() {
param subscription : kind == "bargains" && category == "tech";
}
Preparation:Monitoring:Implementations: IBM InfoSphere Streams (SPL)
• Example:
• Effect: Stream of bargains in tech sector is sent (exported) from
current application, making it available to another application which
imports the stream to monitor bargains in various sectors
epts event processing technical society
76
RA:Preparation:Enrichment:
Enrichment: add some information based on prior events
– E.g. customer purchase: add the customer history to the
purchase event
– E.g. enrich the event with semantic background knowledge
76
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
77
Preparation:Enrichment:Classification
• Alternative Names:
– Event Annotation, Semantic Event
• EPTS Reference Architecture:
– "During event preparation, events may be "enriched"
through knowledge gained through previous events or
data (including semantic background knowledge). "
• EPIA:
– Enrich (EPA): a translate EPA that takes a single input
event, uses it to query data from a global state element,
and creates a derived event which includes the attributes
from the original event, possibly with modified values... 77
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
78
Preparation:Enrichment:Description
• Role / Explanation
– Updating some attribute or metadata of the event with some
other values, possibly computed, from some other event or
data
• Associated Business Rule Specification
– a selection (to be enforced during the processing of events):
All <event entities> that are also <enriched data
expression>
must have <some status>.
78
event Enrichment
f(event) event
Other data
or event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
79
Preparation:Enrichment:Structure
• EPTS Glossary comparison
– Enrichment can be specified in an event pattern (pattern
definition containing additional meta data, semantic
knowledge)
– .. as part of an event processing rule (A method for
enriching events during processing, e.g., by analysing the
meta data and querying external data sources or
inferencing semantic knowledge bases)
– .. or as part of event stream processing (additional
additional information to event stream data, e.g. by
querying external data sources)
79
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
80
Preparation:Enrichment:Implementations
• TIBCO, Oracle, Prova and IBM implementations for
this pattern
80
event Enrichment
f(event) event
Other data
or event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
81
Preparation:Enrichment:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event, data
<if> event-data-match
<then> update-event-with-data
• Effect: find appropriate data for
events and apply enrichment
operations on them*
81
event
Identify
enrichment data
Event
Enrich
Operation
on event
Enrich
data
* note: event object: events cannot be modified
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
82
Preparation:Enrichment:Implementations: Oracle EP: queries
• General pattern template:
SELECT <property>
FROM <STREAM>[WIN],<RELATION>
WHERE <join-condition>
• Effect: enrich events from stream
with contextual data from relation
where condition applies.
82
event
join-condition
event
Operation on
event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
83
Preparation:Enrichment:Implementations: Oracle EP: model
83
• Scenario:
– For every stock event from stock tick stream whose bid is
greater than 5.0, find its company’s location.
• Application Model
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
84
Preparation:Enrichment:Implementations: Oracle EP: queries
84
SELECT event.symbol, location
FROM
StockTickStream [NOW] AS event,
StockTable AS data
WHERE event.symbol = data.symbol AND
event.lastBid > 5.0
No need for window
operator on a ‘relation’
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
85
Preparation:Enrichment:Implementations: Oracle EP: table metaphor
85
Time Input (StockTickStream) Output (QUERY)
1
2
3
Symbol Full-name Location
AAA The AAA company San Francisco
BBB The BBB company San Jose
{“AAA”, 10.0, 10.5} {“AAA”, “San…”}
{“BBB”, 11.0, 12.5} {“BBB”, “San Jose”}
{“AAA”, 4.0, 4.5}
StockTable
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
86
Preparation:Enrichment:Implementations: Prova: rules
• General pattern template: rcvMsg / rcvMult <Event Msg Pattern(s)> :-
<query built-in(s)>,
<enrichment condition(s)>
... .
<logical enrichment rule 1> :-
<enrichment operations>.
• Effect: query data or knowledge from
external source and enrich the event with it
86
event
Query
event*
Operation on
enriched event
External
data
Enrichment
rules
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
87
• Java ..., L=java.util.ArrayList(),, ..., java.lang.String(“Hello”)
• File Input / Output ..., fopen(File,Reader), ...
• XML (DOM) document(DomTree,DocumentReader) :- XML(DocumenReader),
• SQL …,sql_select(DB,cla,[pdb_id,“1alx“],[px,Domain]).
• RDF ...,rdf(http://...,"rdfs",Subject,"rdf_type","gene1_Gene"),
• XQuery ..., XQuery = ' for $name in StatisticsURL//Author[0]/@name/text() return $name', xquery_select(XQuery,name(ExpertName)),
• SPARQL ...,sparql_select(SparqlQuery,name(Name),class(Class), definition(Def)),
Preparation:Enrichment:Implementations: Prova: External Data and Object Integration
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
88
Preparation:Enrichment:Implementations: Prova: Example with SPARQL Query % Filter for car manufacturer stocks and enrich the stock tick
event with data from Wikipedia (DBPedia) about the manufacturer
and the luxury cars
rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-
carManufacturer(S,Man), % filter car manufacturers
findall([luxuryCar|Data],luxuryCar(Man,Name,Car),Data), % query
EnrichedData = [S,Data], % enrich with additional data
sendMsg(SID2,esb,"epa1", inform, happens(tick(EnrichedData,P),T).
% rule implementing the query on DBPedia using SPARQL query
luxuryCar(Manufacturer,Name,Car) :-
Query="SELECT ?manufacturer ?name ?car % SPARQL RDF Query
WHERE {?car <http://purl.org/dc/terms/subject>
<http://dbpedia.org/resource/Category:Luxury_vehicles> .
?car foaf:name ?name .
?car dbo:manufacturer ?man .
?man foaf:name ?manufacturer.
} ORDER by ?manufacturer ?name“,
sparql_select(Query,manufacturer(Manufacturer),name(Name),car(Car)).
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
epts event processing technical society
89
Preparation:Enrichment:Implementations: IBM WODM Decision Server Events
89
event
Enriched
event
• Example: Define a Mapped Key expression, where a
field in the event is used to look up objects in a
database table
• Effect: Required enrichment data is obtained from the
data base and added to the event
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
Mapped Key
look up
epts event processing technical society
90 90
Event Preparation Identification, Selection, Filtering,
Monitoring, Enrichment
/* VWAP = volume-weighted average price */
stream<rstring sym, float32 vwap> Vwaps = PrepEnr_VwapsSource() {}
stream<rstring sym, float32 price> Trades = PrepEnr_TradesSource() {}
stream<rstring sym, float32 vwap, float32 price> E = Join(Vwaps; Trades) {
window
Vwaps : sliding, count(0);
Trades : sliding, count(100), count(1), partitioned;
param
equalityLHS : Vwaps.sym;
equalityRHS : Trades.sym;
partitionByRHS : Trades.sym;
}
Preparation:Enrichment:Implementations: IBM InfoSphere Streams (SPL)
stream
Join Enriched Stream
Trades Trades
enriched with
VWAPs
• Example:
stream VWAPs
epts event processing technical society
91
Event Analysis is the process of analysing suitably
prepared events and their payloads and metadata for
useful information.
For example, event analysis may involve the
identification of events against existing entities.
91
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
EP Patterns:RA:Analysis
epts event processing technical society
92
RA:Analysis:Analytics
Analytics: use of mathematical methods like statistics to
derive additional information about an event or set of
events
– E.g. standard deviation of the response time of a system,
used to give a quantitative comparison of current with past
performance of a system
92
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
93
Analysis:Analytics:Classification
• Alternative Names:
– Event Statistics
• EPTS Reference Architecture:
– “Event Analytics are the use of statistical methods to
derive additional information about an event or set of
events.”
• EPIA:
– Aggregate (EPA): a transformation EPA that takes as
input a collection of events and creates a single derived
event by applying a function over the input events.
93
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
94 event
event
Analysis:Analytics:Description
• Role / Explanation
– Analytics is where multiple data items (in a single event, or
more commonly across multiple events), are analysed
mathematically to define some additional information
– For example: an average or mean value of event data can
be used to compare against subsequent events to indicate
some trend (see Learning)
• Associated Business Rule Specification
– an event analytics (event processing) process:
The <analytic fact> is provided by <analytic function>
applied to <events>
94
event Analytics
f(events) info
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
95
Analysis:Analytics:Structure
• EPTS Glossary comparison
– Cause: An event A is a cause of another event B, if A had to happen
in order for B to happen.
– Event abstraction: The relationship between a complex event and the
other events that it denotes, summarizes, or otherwise represents.
– Derived event (synthesized event, synthetic event): An event that is
generated as a result of applying a method or process to one or
more other events.
– Relationships between events: Events are related by time,
causality, abstraction, and other relationships. Time and causality
impose partial orderings upon events.
– Event pattern: A template containing event templates, relational
operators and variables.
– Constraint (event pattern constraint): A Boolean condition that must
be satisfied by the events observed in a system. 95
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
96
Analysis:Analytics:Implementations
• TIBCO, Prova, Oracle, IBM implementations for this
pattern
96
event Analytics
f(events,data) info
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
past
event data
epts event processing technical society
97
Analysis:Analytics:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event,
historic-event-object
<if>
event-matches-statistical-
requirement
<then>
apply-update-function for
event, historical-event-object
• Effect: event-driven statistic function (eg update of mean
for a given event type or other condition) 97
event
Statistical
operation
or process
Information
past data
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
98
Analysis:Analytics:Implementations: Prova: Rules
• General pattern:
rcvMsg / rcvMult <Event Msg Pattern(s)> :-
<query (past) data>,
<analytics functions(s)
with data>
... .
<analytics rule 1> :-
<Java call to analytics API>,
<or Prova functional programming>.
• Effect: event-driven statistic function (eg update of mean
for a given event type or other condition) 98
event
Analytics
function(s)
Information
(past) data
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
99
Analysis:Analytics:Implementations: Prova: Java Programming and Functional Programming
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
Prova Dynamic Java Programming
• Ability to embed Java calls … :- X = java.lang.Math.abs(2.99), X+1, …
• Java constructors, object instance + methods and static methods, and
public field access; (+ Java exception handling)
Prova Functional Programming
• single‐ and multi‐valued functions,
• functional composition with the extended derive built‐in;
• partial evaluation;
• lambda functions;
• monadic functions;
• monadic bind using a composition of map and join;
• maybe, list, state, tree and fact monads
epts event processing technical society
100
Analysis:Analytics:Implementations: Oracle EP: queries
• General pattern:
SELECT analytical-
function(<property>)
FROM <SOURCE>
WHERE <predicate-condition>
• Effect: performs analytical function on
selected properties, such as std-dev,
regressions, etc.
100
event
analytics
info
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
101 101
Analysis:Analytics:Implementations: IBM WODM Decision Server Events
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
• General Pattern: Filters in Decision Server
Events allow analytic expressions to be applied
epts event processing technical society
102 102
Analysis:Analytics:Implementations: IBM InfoSphere Streams (SPL)
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
stream<rstring sym, float32 vwap, float32 price> E = Join(Vwaps; Trades) {
/* (see Preparation:Enrichment) */
}
/* bargain if price lower than volume-weighted average price */
stream<rstring sym, float32 index> Bargains = Functor(E) {
output Bargains : index = (vwap <= price)
? 0f : price * exp(vwap – price);
}
event stream Functor event
stream
Trades with ave price
Analyze trades to
detect bargains Bargains
• Example:
epts event processing technical society
103
RA:Analysis:Transforms
Transforms: processes carried out on events’ payloads or
data, such for as standardising schema or information
types
– E.g. an insurance application event from a custom IT system
may have the customer information translated into an
ACORD-type structure for onward processing
103
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
104
Analysis:Transforms:Classification
• Alternative Names:
– Event Processes, Event Operations
• EPTS Reference Architecture:
– “Event Transforms are processes carried out on event
payloads or data, either related to event preparation,
analysis or processing.”
• EPIA:
– Transformation (EPA): an EPA that includes a derivation
step and optionally also a filtering step.
• Derived event: an event generated as a result of EP in EPA
104
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
105
Analysis:Transforms:Description
• Role / Explanation
– Transforms are applied to events (event data) to process
into new, derived events or event data
– For example: a transform may reformat an input data
schema into an output schema format
• Associated Business Rule Specification
– an event transform:
The <derived fact> is derived from <input event> when
<transform fn> is applied
105
event Transform
f(event) derivation
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
106
Analysis:Transforms:Structure
• EPTS Glossary comparison
– Derived event (synthesized event, synthetic event): An event that is
generated as a result of applying a method or process to one or
more other events.
106
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
107
Analysis:Transforms:Implementations
• TIBCO, Prova, Oracle, and IBM implementations for
this pattern
107
event Transform
f(events,data) info
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
108
Analysis:Transforms:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event
<if>
event-matches-selection
<then>
apply-derivation-function
for event
• Effect: (usually identified, filtered) events have some
transformation applied to their data, used to update some
event object (or create a new event) 108
event
Event operation
Information
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
109
Analysis:Transforms:Implementations: Prova: Rules
• General pattern:
rcvMsg / rcvMult <Event Msg
Pattern(s)> :-
<transformation function
(Output, Input)>,
... .
<transformation rule 1>(Output,
Input) :-
<transformation logic >
• Effect: transforms selected properties into
new properties, applying function if needed. 109
event
transformation
info
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
110
Analysis:Transforms:Implementations: Oracle EP: queries
• General pattern:
SELECT f(<property>) as <derived>
FROM <SOURCE>
WHERE <predicate-condition>
• Effect: transforms selected properties
into new properties, applying function if
needed.
110
event
transformation
info
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
111 111
Analysis:Transforms:Implementations: IBM WODM Decision Server Events
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
• Example: Use
Javascript to
transform an
event e.g.
transform order
total from number
of units to overall
price in euros
• Transforming in
the event runtime
simplifies coding
in the connectors
epts event processing technical society
112 112
Analysis:Transforms:Implementations: IBM CICS Transaction Server for z/OS
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
• Example: Define an EP
adapter for a CICS Event
Binding (i.e. group of related
events) which will transform
events into a specified format
epts event processing technical society
113 113
stream<rstring sym,float32 price,float32 vol> PreVwaps=Aggregate(Trades) {
/* (see Detection:Aggregation) */
}
/* VWAP = volume-weighted average price */
stream<rstring sym, float32 vwap> Vwaps = Functor(PreVwaps) {
output Vwaps : vwap = priceVol / vol;
}
Analysis:Transforms:Implementations: IBM InfoSphere Streams (SPL)
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
event stream Functor event
stream
Trades Weighting
transformation Trades with price
transformed to volume-
weighted average price
• Example:
epts event processing technical society
114
RA:Analysis:Tracking
Tracking: use of events to follow some related entity’s state
(such as in space, time or process status)
– E.g. “track and trace” of the location of airline baggage
114
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
115
Analysis:Tracking:Classification
• Alternative Names
• EPTS Reference Architecture:
– “Event Tracking is where events related to some entity are
used to identify state changes in that entity.”
• Typically state is related to geolocation, business process, etc
• EPIA:
– State-oriented context: see also spatial context,
segmentation-orientated context, temporal context
115
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
116
Analysis:Tracking:Description
• Role / Explanation
– Tracking is applied to entities via events (event data) to
maintain information about the entity state (eg location).
– For example: a patient may be tracked by active and
passive sensors creating movement and activity events in a
care facility
– Usually associated with the term “track and trace”
• Associated Business Rule Specification
– an track operation:
The <entity> is tracked to <state>
whenever
<event> occurs 116
event Tracking
f(event)
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
entity
Updated
entity
epts event processing technical society
117
Analysis:Tracking:Structure
• EPTS Glossary comparison
– Derived event (synthesized event, synthetic event): An event that is
generated as a result of applying a method or process to one or
more other events.
117
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
118
Analysis:Tracking:Implementations
• TIBCO, Prova, Oracle, IBM implementations for this
pattern
118
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
event Tracking
f(event)
entity
Updated
entity
epts event processing technical society
119
Analysis:Tracking:Implementations: TIBCO BusinessEvents:StateModel
• General pattern: <state models for concept C>
<route TrackingState n to n+1>
<on> event
<where> event-filter
• Effect: change tracking state
for some concept / entity
• Notes:
1. this is best for explicit, unchanging states
(eg “in range” vs “out of range”)
2. “Tracing” can be added with a simple additional action… 119
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
event
Track(event)
New State
entity
epts event processing technical society
120
Analysis:Tracking:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event, entity
<if>
event-matches-entity-selection
<then>
apply-event-tracking-data
to entity
• Effect: tracking events are associated
to the relevant entity (eg by Rete algorithm)
and the rule updates the entity track
• Note: also updateable to “trace” with an additional action
120
event
Event tracking
conditions
Update entity track
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
entity
epts event processing technical society
121
Analysis:Tracking:Implementations Prova: Rules
121
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
event Tracking Rule
f(event)
entity
Updated
entity
Java
Type Semantic
Type
epts event processing technical society
Prova: Semantic Type System Top-Level Ontologies
General concepts such as space, time, event, action and their properties and relations
Temporal Ontology
Action Ontology Process
Ontology Agent
Ontology Situation Ontology
Domain Ontologies
Vocabularies related to specific domains by specializing the concepts introduced in the top-level ontology
Task Activities Ontologies
Vocabularies related to generic tasks or
activities by specializing the
concepts introduced in the top-level ontology
Application Ontologies
Specific user/application ontologies
E.g. ontologies describing roles played by domain entities while perfoming application activities
Spatio Ontology
Event Ontology
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
Prova: Situation/State Top Ontology Model
Situation
Heterogeneous Situation
Homogenous Situation
Dynamic Change
Situation
Time Frame
Situation
Frequency Situation
State Situation
Process Situation
Iterative Situation
Habitual Situation
Situation Properties
(relations to time, location, participants, …)
Situation Content
hasContent hasProperties
Situation Types
Situation Descriptions
LaysOn TheFloor
TrafficLightChanges
Within5Minutes
He Runs
Rings3 Times
He Coughs
He Smokes
Situation Individuals
use the other top ontologies
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
124
Analysis:Tracking:Implementations: Oracle EP: EPN
124
• Effect:
– Queries select events to be tracked by putting them in
cache. Events can be retrieved by using cache as source
to processor.
• Application Model
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
epts event processing technical society
125
type UpdateKind = enum { MOVE, ENTER }; /* ... */ stream<int32 vid, UpdateKind kind> Updates = Custom(Tracked as Curr) { logic state : { mutable map<int32, Position> _vid2pos = { }; } onTuple Tracked : { if (Curr.vid in _vid2pos) { if (Curr.pos != _vid2pos[vid]) submit({ vid=Curr.vid, kind=MOVE }, Updates); } else { submit({ vid=Curr.vid, kind=ENTER }, Updates); } _vid2pos[Curr.vid] = Curr.pos; } }
Analysis:Tracking:Implementations: IBM InfoSphere Streams (SPL)
Event Analysis Analytics, Transforms, Tracking,
Scoring, Rating, Classification
• Example:
• Effect: Custom operator used to Track position of vehicles
(by id) in transportation application
epts event processing technical society
126
RA:Analysis:Scoring
Scoring: ranking events or event data based on some
predefined criteria
– E.g. scoring a customer account application based on
weighting specific aspects of the application to give an
overall “score”
– See Score Modeling and Predictive Analytics
126
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
127
Analysis:Scoring:Classification
• Alternative Names
• EPTS Reference Architecture:
– “Event Scoring is the process by which events are ranked
using a score, usually as a part of a statistical analysis of a
set of events”.
– Note: Possibly this should be subsumed into Event Analytics
• EPIA:
• Aggregate (EPA): a transformation EPA that takes as input a
collection of events and creates a single derived event by
applying a function over the input events.
127
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
128
Analysis:Scoring:Description
• Role / Explanation
– Scoring is applied to events or entities by associating score
values to attributes of the event or entity.
– For example: an intelligence report event may be scored
against its attributes (source, time, subject etc) to give an
overall indication (score, for example as a %) of its reliability
– Usually associated with predictive analytics
• Associated Business Rule Specification
– an score operation on an event:
The <entity> is given a score <s> by applying
transform
model <m>
128
event Score model
f(event)
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
Score
epts event processing technical society
129
Analysis:Scoring:Structure
• EPTS Glossary comparison
– Derived event (synthesized event, synthetic event): An event that is
generated as a result of applying a method or process to one or
more other events.
– Relationships between events: Events are related by time,
causality, abstraction, and other relationships. Time and causality
impose partial orderings upon events.
– Event pattern: A template containing event templates, relational
operators and variables.
– Constraint (event pattern constraint): A Boolean condition that must
be satisfied by the events observed in a system.
129
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
130
Analysis:Scoring:Implementations
• TIBCO, Prova, Oracle, and IBM implementations for
this pattern
130
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
event Score model
f(event) Score
epts event processing technical society
131
Analysis:Scoring:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event
<if>
event-matches-entity-selection
<then>
apply-score-model(eventdata),
combine-score-criteria
• Effect: events’ criteria are applied to
score functions and then combined into a
single score via an event object
• Note: score typically applied in a separate rule
(if score>50% then …) 131
event
Score model
(event)
Update score
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
132
Analysis:Scoring:Implementations: Prova: Rules
• General pattern: rcvMsg / rcvMult <event> :-
@score(Old) <event>,
<score function (event, Old, New),
<update> @score(New) <event>.
<score function rule>(E,O,N) :-
<compute new score>.
• Effect: events’ metadata score (and other
metadata or event data properties) are
applied to score functions and then combined into a
single updated score as metadata of the event object
132
event
Score model
(event)
Update
event
score
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
133
Analysis:Scoring:Implementations: Prova: Example – Metadata Scoped Reasoning
% event instances with metadata @key(value,[,value]*)
@score(1) @src(stream1) @label(e1) tick("Opel",10,t1).
@score(3) @src(stream2) @label(e2) tick("Opel",15,t1).
happens(tick(S,P),T):-
% scope defined over @score metadata bound to "Value"
@score(Value) tick(S,P,T) [Value>2]. %guard Value > 2
@score(1) @src(stream1) @label(e1) tick("Opel",10,t1).
@score(3) @src(stream2) @label(e2) tick("Opel",15,t1).
happens(tick(S,P),T):-
% select ticks from „stream1“
@score(Value) @src(stream1) @label(E) tick(S,P,T),
NewValue = Value + 3, %add 3 to score „Value“
%update selected events from „stream1“ with „NewValue“
update(@label(E), @score(NewValue) tick(S,P,T)),
@score(S) tick(S,P,T) [S>3]. %guard Value > 3
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
134
Analysis:Scoring:Implementations: Oracle EP: queries
• General pattern:
SELECT score@model(<property>)
FROM <SOURCE>
WHERE <predicate-condition>
• Effect: scores properties of events using
model.
134
event
score
info
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
135
RA:Analysis:Rating
Rating: comparison of events or event data with some metric
to provide an ordering
– E.g. trade transactions may be rated with respect to overall
achieved profit
135
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
136
Analysis:Rating:Classification
• Alternative Names
• EPTS Reference Architecture:
– “Event Rating is where events are compared to others to
associate some importance or other, possibly relative,
measurement to the event”.
• EPIA:
– Transformation (EPA): an EPA that includes a derivation
step and optionally also a filtering step.
• Derived event: an event generated as a result of EP in EPA
136
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
137
Analysis:Rating:Description
• Role / Explanation
– Rating provides a quantitative or ordering.
– For example: an customer order shipment may be rated as
“courier” based on the customer order details
– Could be part of “classification”
• Dictionary definition: classification according to grade or rank
• Associated Business Rule Specification
– an rating operation on an event:
The <event> is rated to <rating> by <rating criteria>
137
event Rating
f(event)
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
Rating
epts event processing technical society
138
Analysis:Rating:Structure
• EPTS Glossary comparison
– Derived event (synthesized event, synthetic event): An event that is
generated as a result of applying a method or process to one or
more other events.
– Relationships between events: Events are related by time,
causality, abstraction, and other relationships. Time and causality
impose partial orderings upon events.
– Event pattern: A template containing event templates, relational
operators and variables.
– Constraint (event pattern constraint): A Boolean condition that must
be satisfied by the events observed in a system.
138
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
139
Analysis:Rating:Implementations
• TIBCO, Prova, Oracle, IBM implementations for this
pattern
139
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
event Rating
f(event) Rating
epts event processing technical society
140
Analysis:Rating:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event
<if>
event-matches-entity-selection
<then>
rating = rating-model(event)
• Effect: events’ criteria
(and possibly enrichment data)
are applied to the rating function
to create a rating classification or ordering
• Note: rating typically applied in a separate rule
(if rating = “First” then …) 140
event
Rating
(event)
Rating
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
141
Analysis:Scoring:Implementations: Prova: Rules
• General pattern:
see scoring
• Effect: rate events and select rated events
141
event
rating
(event)
Rated Event
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
142
Analysis:Rating:Implementations: Prova: Metadata Scoped Reasoning and Guards % stream1 is trusted but stream2 is not, so one
solution is found: X=e1
@src(stream1) event(e1).
@src(stream2) event(e2).
%note, for simplicity this is just a simple fact, but
more complicated rating, trust, reputation policies
could be defined
trusted(stream1).%only event from „stream1“ are trusted
ratedEvent(X):-
@src(Source) %scoped reasoning on @src
event(X) [trusted(Source)]. %guard on trusted sources
:-solve(ratedEvent(X)). % => X=e1 (but not e2)
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
143
Analysis:Rating:Implementations: Oracle EP: queries
• General pattern:
SELECT rate@model(<property>)
FROM <SOURCE>
WHERE <predicate-condition>
• Effect: rates properties of events using
model.
143
event
rate
info
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
144 144
Analysis:Rating:Implementations: IBM WODM Decision Server Events
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
• General Pattern: Rating in WODM is typically
carried out by the Decision Server Rules
component – the output events indicate something
interesting happened, and then business rules rate
the type of response depending on details in the
events
epts event processing technical society
145
type Bracket = enum { YOUNG, OLD, OTHER };
stream<rstring name, int32 age> Folks = /*...*/
stream<rstring name, Bracket bracket> Rated = Functor(Folks) {
output Rated : bracket = age < 30 ? YOUNG : age > 65 ? OLD : OTHER;
}
Analysis:Rating:Implementations: IBM InfoSphere Streams (SPL)
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
event stream Functor
event stream
Folks Rating functor
to rate by age
• Example:
Events rated as
YOUNG, OLD or
OTHER
epts event processing technical society
146
RA:Analysis:Classification
Classification: comparison with and association of events with
some classification scheme to which the event is applied
– E.g. classifying a credit card user as a “gold shopper” for
marketing purposes
146
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
147
Analysis:Classification:Classification
• Alternative Names
• EPTS Reference Architecture:
– “Event Classification is where events are associated with
some classification scheme (for use in downstream
processing)”.
• EPIA:
– Transformation (EPA): an EPA that includes a derivation
step and optionally also a filtering step.
• Derived event: an event generated as a result of EP in EPA
147
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
148
Analysis:Classification:Description
• Role / Explanation
– Classification provides associates an event to a class or
category.
– For example: an customer may be classified as “Gold
customer” based on the customer order details
– Discovery / learning may be involved in the definition of
classifications
• Associated Business Rule Specification
– an event is classified:
The <event> is classified as <class>
by <classification criteria>
148 event
Classification
f(event)
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
Classification
epts event processing technical society
149
Analysis:Classification:Structure
• EPTS Glossary comparison
– Derived event (synthesized event, synthetic event): An event that is
generated as a result of applying a method or process to one or
more other events.
– Relationships between events: Events are related by time,
causality, abstraction, and other relationships. Time and causality
impose partial orderings upon events.
– Event pattern: A template containing event templates, relational
operators and variables.
– Constraint (event pattern constraint): A Boolean condition that must
be satisfied by the events observed in a system.
149
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
150
Analysis:Classification:Implementations
• TIBCO, Prova, Oracle, IBM implementations for this
pattern
150
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
event Classification
f(event) Classification
epts event processing technical society
151
Analysis:Classification:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event
<if>
event-matches-classification
<then>
apply-classification(event)
• Effect: 1 rule per criteria + classification,
and then classification is updated as an
event object / concept attribute
• Note: classification typically used in separate rules
(if class = “Gold customer” then …)
151
event
Classification
(event)
Classification
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
152
Analysis:Classification:Implementations: Prova: Rules
• General pattern: <rcvMsg / rcvMult> Event ^^ Semantic Type :-
<conditional type discovery>,
derivedEvent ^^ Semantic Type.
• Effect: classify event according to a (semantic) type system.
Discover event type (may include learning). 152
Event (Semantic)
Classification
f(event)
Typed
Event
(Semantic)
Type System
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
153
Analysis:Classification:Implementations: Prova: Rules - Example
Event Stream {(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) }
{(OPEL, is_a, Car_Manufacturer), (Car_Manufacturer, sameAs, Motor_Vehicel_Manufacturer),
(Car_Manufacturer, subClassOf, Automotive_Company), (Automotive_Company, sameAs, Automotive_Corporation), (Automotive_Company, subClassOf, Automotive_Industry),
(Automotive_Corporation, subClassOf, Corporation) (Major_Corporation, have, over_10,000_employees),
(Major_Corporation, subClassOf, Corporation)}
Semantic Knowledge Base
(T-Box Model is used as Type
System)
rcvMult(SID,stream,“S&P500“, inform,
tick(Name^^Car:Manufacturer,P,T)) :-
size(Name, Eymployees), Employees > 10000, %type discovery
sendMsg(SID,self,0,inform,tick(Name^^Major_Cooperation,P,T)
.
Classification according to
Type System
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
Classification by Type Discovery
(may also include Type Learning)
epts event processing technical society
154
Analysis:Classification:Implementations: Oracle EP: queries
• General pattern:
SELECT classify@model(<property>)
AS <classification>
FROM <SOURCE>
WHERE <predicate-condition>
• Effect: classifies properties of events
using model, and applies to result.
154
event
classification
info
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
155
Analysis:Classification:Implementations: IBM WODM Decision Server Events
155
event
condition
output event (action)
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
classification rules
decision
• General Pattern: Decision Server
Events component produces actions
(or output events) and then
business rules in Decision Server
Rules apply classification to these to
decide on the final outcome
epts event processing technical society
156 156
/* (see Analysis:Rating) */
Analysis:Classification:Implementations: IBM InfoSphere Streams (SPL)
Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification
epts event processing technical society
157 157
Complex Event Detection Consolidation, Composition,
Aggregation
Complex Event Detection is the process by which
event analysis results in the creation of new event information,
or the update of existing complex events.
For example, complex event detection may result
from identifying a pattern of event occurences
that indicate some complex event has taken or is taking place.
EP Patterns:RA:Detection
epts event processing technical society
158
RA:Detection:Discussion
• Are consolidation, composition and aggregation
sufficiently defined or different?
• Original EPTS RA definitions
– Consolidation: combining disparate events together into a
"main" or "primary" event.
– Composition: composing new, complex events from existing,
possibly source, events
– Aggregation: combining events to provide new or useful
information, such as trend information and event statistics.
158
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
159
RA:Detection:Discussion(2)
• EPTS Glossary (v2) Terms
– Derived event (synthesized event, synthetic event)
• An event that is generated as a result of applying a method or
process to one or more other events.
– Composite event
• A (type of) derived event that is created by combining a set of
other simple or complex events (known as its members) using a
specific set of event constructors such as disjunction, conjunction,
and sequence. A composite event always includes the member
(base)events from which it is derived.
• A derived event is not a composite if its method of derivation lies
outside a specified set of allowed constructors.
159
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
160
RA:Detection:Discussion(3)
• Dictionary definitions
– Consolidation: bring together (separate parts) into a single
or unified whole
– Composition: combining parts or elements to form a whole
– Aggregation: a group or mass of distinct or varied things
• From these we can infer:
composition is a generic term,
consolidation joins events of a similar or related type,
and
aggregation joins events of differing types
160
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
161
RA:Detection:Discussion(4)
• Other interesting pattern classifications?
– Composition from an ordered or partially ordered sequence of
events
– Composition from missing events or non-events
– Composition from out of order events in some sequence of
events
• Other types of complex event detection?
– Situations or States, indicating some prior sequence of
events, significant to be classified or specified
161
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
162
RA:Detection:Discussion(5)
• Summary
– Original EPTS RA classification can be safely revised /
extended …
– Here we look at
Consolidation and Aggregation (i.e. composition of similar
and different event types)
+ Composition of events including a non-event
162
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
163
RA:Detection:Discussion(6)
• Possible new EP RA definitions:
– Consolidation: reinforcing a primary event with evidence
from new events of the same type
– Composition: composing a new event from existing events
of similar types + all contributing events are included
– Aggregation: combining event data from disparate event
types to create a new event
163
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
164
RA:Detection:Consolidation
Consolidation: combine events of similar type together into an new event
or combine disparate events together into a main or primary event
– E.g. multiple events occur to indicate a complex event; the complex
event is given a new consolidate identity as derived primary event or
main event
– E.g. several messages occur that together form a complex event
indicating a business transaction
– E.g. multiple events of the same type occur within a time period to
indicate an event called “a high rate of events”
– E.g. abstraction of events into a situation which is initiated or terminated
by the (derived complex) event as effect of the detection and
consolidation of the complex event, e.g. the complex event “plane take-
off” has the effect initiating the situation “flying”
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
165
Detection:Consolidation:Classification • Alternative Names:
– Event Reinforcement from Subevents, Event / Situation Reasoning
• EPTS Reference Architecture:
– "During complex event detection, combining events together into a
"main" or "primary" event. Similar to event aggregation and
composition", but with a new explicit derived event from the
aggregated/composed events (the consolidated derived event might
not contain all events from which it was derived);
– a special abstraction of the effect of a (derived) event is a situation
which is initiated or terminated by events
• EPIA:
– Aggregate (EPA): a transformation EPA that takes as input a
collection of events and creates a single derived event by applying a
function over the input events. 165
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
166 event
event
Detection:Consolidation:Description
• Role / Explanation
– Consolidation is used to describe several events supporting
the creation of a single composite event, generally where
component events together provide evidence.
– For example: a “conflict” event (effect = situation “in
conflict”) is considered to take place when both a “conflict
declaration” occurs and an “act of violence” occurs.
• Associated Business Rule Specification
– a consolidation (to be enforced during event processing):
The <collection of events> supporting <event> that
are <related by
some relationship
constraint>. 166
event Consolidation
f(events) event
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
167
Detection:Consolidation:Structure
• EPTS Glossary comparison
– The term consolidated event is sometimes used for some
forms of composite or derived event.
– Composite event: a derived, complex event
• is created by combining base events using a specific set of
event constructors such as disjunction, conjunction,
sequence, etc.
• always includes the base (member) events from which it is
derived.
– Derived event (/synthesized event): an event that is
generated as a result of applying a method or process to
one or more other events.
167
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
168
Detection:Consolidation:Implementations
• TIBCO, Prova, Oracle, IBM implementations for this
pattern
168
Complex Event Detection Consolidation, Composition,
Aggregation
event event
event Consolidation
f(events)
derived
event
epts event processing technical society
169
Detection:Consolidation:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event,
consolidated-event-object
<if>
event-supports-event-object
<then>
update-consolidated-object
with event-data
• Effect: construct evidence for
(main event) object for n incoming events 169
event
Associate
event for
consolidation
Updated
Main Event
Object
Consolidate
/ reinforce
event
Main event
object
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
170
Detection:Consolidation:Implementations: Prova: Consolidation
• General pattern:
<rcvMult> Event :-
DerivedEvent = <consolidate> (
<compose/aggregate> Events ),
<initiate / terminate> (
DerivedEvent, Situation).
• Effect: a detected complex event (see
composition, aggregation) is given a new
derived event identity. The effect of the
event occurrence / detection might initiate of
terminate a situation. 170
Event
Composition/Aggregation
f(Events)
derived
event
Event
Complex Event Detection Consolidation, Composition,
Aggregation
Consolidate
f(Events)
Situation Situation
initiate terminate
epts event processing technical society
171 171
– Interval-based Event Calculus semantics (model-theory + proof theory) based on
time intervals modeled as fluents
I: Tinterval x Fl {true, false}
– Example: D = A;(B;C) (consolidation: derive event D from sequence composition)
– Example: derived situation from complex event detection (consolidation: initiate situation1 by event D)
initiates(D,situation1,T). holdsAt(situation1, t5)? => yes
T1 T2 T3 T4
A B C
[B,C] [A,A]
[[A,A],[B,C]]
Complex Event Detection Consolidation, Composition,
Aggregation
Detection:Consolidation:Implementations: Prova: Consolidation Example
D = [A,C]
Composition
Consolidation
epts event processing technical society
172
Dection:Consolidation:Implementations: Oracle EP: pattern matching
• General pattern:
SELECT <property>
FROM <SOURCE>
MATCH_RECOGNIZE (
MEASURES <property>
PATTERN (<regular-expression>)
DEFINE <expression>
)
• Effect: consolidates events from source
into measures. 172
Complex Event Detection Consolidation, Composition,
Aggregation
Event
Pattern match
derived
event
Event
epts event processing technical society
173
Dection:Consolidation:Implementations: Oracle EP: pattern matching
• Example:
SELECT M.goingUpPrice
FROM marketFeed
MATCH_RECOGNIZE (
MEASURES B.price as goingUpPrice
PATTERN (A B)
DEFINEA as price < 29.0,B as price > 30.0
) as M
• Description: consolidates stock ticks
into a single ‘going up’ alert event. 173
Complex Event Detection Consolidation, Composition,
Aggregation
Event
Pattern match
derived
event
Event
epts event processing technical society
174 174
Detection:Consolidation:Implementations: IBM WODM Decision Server Events
Complex Event Detection Consolidation, Composition,
Aggregation
• Example: LargeTransaction events generated for any
type of large banking transaction (deposit, withdrawal,
transfer)
• RecentSignificantAccountActivity is consolidation of
several recent LargeTransaction events, each such
event further reinforces the consolidated event
• Effect of the example: Detect customers with
recent significant account activity
Consolidate
Consolidated
Event
event event
Large Check
Deposit Large Cash
withdrawal
Significant
Account
Activity
Note: this screenshot is from an older version of the product
epts event processing technical society
175 175
type Trade = tuple<rstring sym, int32 price, int32 vol>;
type Quote = tuple<rstring sym, int32 offer>;
type TradeOrQuote =
tuple<rstring sym, int32 price, int32 vol, enum{TRADE, QUOTE} kind>;
/*...*/
stream<Trade> Trades = /*...*/
stream<Quote> Quotes = /*...*/
stream<TradeOrQuote> TQs = Custom(Trades; Quotes) {
logic
onTuple Trades:
submit({sym=sym, price=price, vol=vol, kind=TRADE}, TQs);
onTuple Quotes:
submit({sym=sym, price=offer, vol=-1, kind=QUOTE}, TQs);
}
Detection:Consolidation:Implementations: IBM InfoSphere Streams (SPL)
Complex Event Detection Consolidation, Composition,
Aggregation
stream Custom Stream
Trades
TQs stream Quotes
epts event processing technical society
176
RA:Detection:Composition
Composition: composing new complex events from existing,
possibly source, events
– E.g. deduce a complex event from a history of past events
Interesting compositions
1. By type (see consolidation and aggregation)
2. By sequence, order, and missing events
3. By combination of operators
(e.g. sequence of increasing values until stops increasing) 176
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
177
Detection:Composition:Classification
• EPTS Reference Architecture:
• "Event composition is about composing new, complex events
from existing, possibly source, events.”
• For example, event composition may take information from a
range of past events to determine that a new, complex, event has
occurred and needs to be composed.
• EPIA:
– Aggregate (EPA): a transformation EPA that takes as input a
collection of events and creates a single derived event by
applying a function over the input events.
177
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
178 event
event
Detection:Composition:Description
• Role / Explanation
– Composition is used to describe combining multiple events
into a composite event, generally where the component
events justify the existence of the complex event.
– For example: a telephone call event is considered to take
place when a call end event occurs after a call start event.
• Associated Business Rule Specification
– a composition (to be enforced during event processing):
The <collection> of <event entities> that
are <related by
some relationship
constraint>.
178
event Composition
f(events) event
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
179
Detection:Composition:Structure
• EPTS Glossary comparison:
– Composite event: A derived event that is created by combining a
set of other simple or complex events (known as its members)
using a specific set of event constructors such as disjunction,
conjunction, and sequence.
– Event pattern: A template containing event templates, relational
operators and variables. An event pattern can match sets of
related events by replacing variables with values.
– Event pattern detection: Finding instances of an event pattern.
– Event template: An event form or descriptor some of whose
parameters are variables. An event template matches single
events by replacing the variables with values. 179
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
180
Detection:Composition:Implementations: Missing Event in a Series
• TIBCO, Prova, Oracle, IBM implementations for this
pattern
180
event event
event
Composition
f(events)
over some
pattern
sequence
Missing event
indicator
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
181
Detection:Composition:Implementations: Missing Event in a Series :TIBCO BusinessEvents: Rules
• General pattern:
• Rules to
1. indicate matching process
is continuing to succeed
2. Indicate end of sequence and therefore an
event is missing
• Effect: construct evidence for the
missing event object based on
n incoming events / target pattern
181
event
Compare
event to
Target Pattern
Missing event
Match end of
sequence
for missing
event
Target pattern
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
182
Detection:Composition:Implementations: Missing Event in a Series :TIBCO BusinessEvents: Rules(2)
• Rule 1
<declare> event, target-event-pattern
<if>
not event-matches-target-event-pattern
<then>
reset-target
• Rule 2
<declare> event, target-event-pattern
<if>
event-matches-target-event-pattern-end
<then>
reset-target, create missing-event-object
182
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
183
Detection:Composition:Implementations: Prova Complex Event Processing
• General pattern: <declare> event,
event-object
<if>
event-relates-to-event-object
<then>
define-composed-object-event
with event-data
• Effect: construct evidence for
(new event) object for n incoming events 183
event
Relate
events
Complex Event
Object
Compose
complex event
Event object
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
184 184
– Interval-based Event Calculus semantics (model-theory + proof theory) based on
time intervals modeled as fluents
I: Ti x Fl {true, false}
– Example: D = A;(B;C) (consolidation: derive event D from sequence composition)
– Example: derived situation from complex event detection (consolidation: initiate situation1 by event D)
initiates(D,situation1,T). holdsAt(situation1, t5). => yes
T1 T2 T3 T4
A B C
[B,C] [A,A]
[[A,A],[B,C]]
Complex Event Detection Consolidation, Composition,
Aggregation
Detection:Consolidation:Implementations: Prova: Composition Example
D = [A,C]
Composition
Consolidation
epts event processing technical society
185
Detection:Composition:Implementations: Prova: Event Composition as Event-Driven Workflow
rcvMsg(XID,Process,From,event,["A"]) :-
fork_b_c(XID, Process).
fork_b_c(XID, Process) :-
@group(p1) rcvMsg(XID,Process,From,event,["B"]), … .
fork_b_c(XID, Process) :-
@group(p1) rcvMsg(XID,Process,From,event,["C"]), … .
fork_b_c(XID, Process) :-
% OR reaction group "p1" waits for either of the two
event message handlers "B" or "C" and terminates the
alternative reaction if one arrives
@or(p1) rcvMsg(XID,Process,From,or,_).
Complex Event Detection Consolidation, Composition,
Aggregation
A
B
C
185
epts event processing technical society
186
Detection:Composition:Implementations: Oracle EP: missing event
• General pattern:
SELECT <property>
FROM <SOURCE>
MATCH_RECOGNIZE (
MEASURES <property>
PATTERN (<regular-expression>)
DURATION <time-interval>
DEFINE <expression>
)
• Effect: detects if pattern has not
matched after duration expires. 186
Complex Event Detection Consolidation, Composition,
Aggregation
Event
Pattern match
derived
event
Event
epts event processing technical society
187
Detection:Composition:Implementations: Oracle EP: missing event
• Example:
SELECT M.maxUpPrice FROM marketFeed
MATCH_RECOGNIZE (
PARTITION BY symbol
MEASURESMAX(B.price) as maxUpPrice
INCLUDE TIMER EVENTS
PATTERN (A B+? C) DURATION 10 MINUTES
DEFINEB as B.price > A.price,
C as C.price < MAX(B.price)
) as M
• Description: send max up price even if trend is
not going down after 10 minutes. In other words,
even if going “down event” is missing. 187
Complex Event Detection Consolidation, Composition,
Aggregation
Event
Pattern match
derived
event
Event
epts event processing technical society
188 188
Detection:Composition:Implementations: IBM WODM Decision Server Events
Complex Event Detection Consolidation, Composition,
Aggregation
• General Pattern: Share data across multiple events
by appending the data to be composed into a business
object array... then when processing the events, use
relevant composition
• Effect of the example: sum up the value of a
customer’s purchases over a week
Add to array
business
Object Array
Compose
Operation
event event
epts event processing technical society
189 189
/* research prototype, see DEBS 2012 paper */
stream<MatchT> Matches = MatchRegex(Quotes) {
param
pattern : ". rise+ drop+ rise+ drop* deep";
partitionBy : symbol;
predicates : {
rise = price>First(price) && price>=Last(price),
drop = price>=First(price) && price<Last(price),
deep = price<First(price) && price<Last(price) };
output
Matches : symbol=symbol, seqNum=First(seqNum),
count=Count(), maxPrice=Max(price);
}
Detection:Composition:Implementations: IBM InfoSphere Streams (SPL)
Complex Event Detection Consolidation, Composition,
Aggregation
• Example:
• Effect: Compose quotes based on price behaviour
epts event processing technical society
190
Detection:Aggregation
Aggregation: combine events of dissimilar type together into
an event
– E.g. combine credit card purchase event with competitor
store nearby not-busy event, and generate competitor offer
event for a complementary product
190
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
191
Detection:Aggregation:Classification
• Alternative Names:
– Event Summarization
• EPTS Reference Architecture:
– "During complex event detection, combining events to
provide new or useful information, such as trend
information and event statistics. Similar to event
consolidation "
• EPIA:
– Aggregate (EPA): a transformation EPA that takes as
input a collection of events and creates a single derived
event by applying a function over the input events. 191
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
192
event event
Detection:Aggregation:Description
• Role / Explanation
– Aggregation is where multiple events support the creation of a
composite event, either as components or evidence, and are of
different type.
– For example: a “start call” event aggregated with an “end call” event
indicates a “telephone call event”
– For example: generate a (composite) event that contains the medium
price of a stock over a 10 minute stream of events.
• Associated Business Rule Specification
– a summarisation (to be enforced during event processing):
The <aggregation fn> of <event entities> that
are <selection constraint>
must have
<some constraint>. 192
event Aggregation
f(events) event
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
193
Detection:Aggregation:Structure
• EPTS Glossary comparison: see previous
193
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
194
Detection:Aggregation:Implementations
• TIBCO, Oracle, Prova and IBM implementations for
this pattern
194
event event
event Aggregation
f(events) event
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
195
Detection:Aggregation:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event,
aggregation-object
<if>
event-member-of-aggregation
<then>
update-aggregation-object
with event-data
• Effect: construct aggregation
(event) object for each incoming event 195
event
Identify
event for
aggregation
Updated
Aggregation
Object
Aggregate
Operation
on event
Aggregation
object
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
196
Detection:Aggregation:Implementations: Oracle EP: queries
• General pattern template:
SELECT <summary-property>
FROM <STREAM>[WIN]
WHERE <predicate>
GROUP BY <aggregation-identity>
• Effect: summarizes the properties of
several simple event into a new
complex event
196
event
Identify
event for
aggregation
Updated
Aggregation
Object
Aggregate
Operation
on event
Aggregation
object
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
197
Detection:Aggregation:Implementations: Oracle EP: model
197
• Scenario:
– Output the average bid and ask price of a stock in the
last 10 seconds.
• Application Model
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
198
Detection:Aggregation:Implementations: Oracle EP: queries
198
SELECT symbol, AVG(bid), AVG(ask)
FROM
StockTickStream [RANGE 10 SECONDS]
GROUP BY
symbol
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
199
Detection:Aggregation:Implementations: Oracle EP: table metaphor
199
Time Input Output
1s
4s
9s
15s
20s
{“AAA”, 10.0, 12.0} {“AAA”, 10.0, 12.0}
{“AAA”, 12.0, 14.0} {“AAA”, 11.0, 13.0}
{“BBB”, 4.0, 5.0} {“AAA”, 11.0, 13},
{“BBB”, 4.0, 5.0}
{“BBB”, 8.0, 10.0} {“BBB”, 6.0, 7.5}
{“BBB”, 8.0, 10.0}
> 10s
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
200
Detection:Aggregation:Implementations: Prova: rules
• General pattern template:
<rule_head> :-
<Create Aggregator>,
@group(<reaction group>)
@timer|size(Start,Interval,Aggregator)
rcvMsg <EventPattern> [<Aggregate Operation>].
<rule_head> :-
@or(<reaction group>)
rcvMsg <Aggregator>,
... <consume Aggregator>.
• Effect: Repeated incremental
aggregations over new events
200
event
Check & Update
Timer / Size Count
Updated
Aggregation
Object
Aggregate
Operation
on event
Aggregation
object
Identify event
for aggregation
Consume
Aggregation
Object
or
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
201
Detection:Aggregation:Implementations: Prova: Example with Time Counter
% This reaction operates indefinitely. When the timer
elapses (after 25 ms), the groupby map Counter is sent
as part of the aggregation event and consumed in or
group, and the timer is reset back to the second
argument of @timer.
groupby_rate() :-
Counter = ws.prova.eventing.MapCounter(), % Aggr. Obj.
@group(g1) @timer(25,25,Counter) % timer every 25 ms
rcvMsg(XID,stream,From,inform,tick(S,P,T)) % event
[IM=T,Counter.incrementAt(IM)]. % aggr. operation
groupby_rate() :-
% receive the aggregation counter in the or reaction
@or(g1) rcvMsg(XID,self,From,or,[Counter]),
... <consume the Counter aggreation object>.
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
202
Detection:Aggregation:Implementations: IBM WODM Decision Server Events
• General Pattern: Share data across multiple events
by appending the data to be aggregated into a business
object array... then when processing the events, use
relevant aggregator
• Effect of the example: sum up the value of a
customer’s purchases over a week
Add to array
business
Object Array
Operation
on aggregate
event event
Complex Event Detection Consolidation, Composition,
Aggregation
epts event processing technical society
203 203
type Trade = rstring sym, timestamp ts, decimal64 price, decimal64 vol;
/* define PreVwap to subsume attributes of Trade */
type PreVwap = Trade, tuple<decimal64 priceVol>;
/* ... */
/* VWAP = volume-weighted average price */
stream<PreVwap> PreVwaps = Aggregate(Trades) {
window Trades : sliding, delta(ts, 60.0), count(1), partitioned;
param partitionBy : sym;
output PreVwaps : priceVol = Sum(price*vol), vol = Sum(vol);
}
Detection:Aggregation:Implementations: IBM InfoSphere Streams (SPL)
Complex Event Detection Consolidation, Composition,
Aggregation
Aggregate stream
Trades Summed trades
• Example:
Sliding Window
epts event processing technical society
204 204
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
Event Reaction is the process subsequent
to event analysis and complex event detection
to handle the results of analysis and detection.
For example, an event reaction could be
the invocation of some service to process the event
in some particular way.
EP Patterns:RA:Reaction
epts event processing technical society
205
RA:Reaction:Assessment:
Assessment: evaluate the event for inclusion in some
process, collection, classification or complex event
– E.g. assess a cash withdrawal event for signs of it being a
fraud event
205
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
206
Reaction:Assessment:Classification
• Alternative Names:
– Event Classification
• EPTS Reference Architecture:
– Event assessment is the process by which an event is
assessed for inclusion in some process, incorporation in
some other event, etc.
• EPIA: no direct analogy, possibly maps to translate
– Translate (EPA): a stateless Transformation EPA that
takes a single event as its input, and generates a single
derived event which is a function of the input event, using
a derivation formula 206
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
207
Reaction:Assessment:Description
• Role / Explanation
– Assessment is used to describe the process of evaluating an
event for the purposes of inclusion in some process, collection
or classification.
– For example: assess a parcel scan event to check whether it is
classed as “in order” or “out of order” relative to some
presumed parcel process
• Associated Business Rule Specification
– a constraint (to be enforced during event processing):
The <event> that
satisfies <assessment constraint>
achieves a
<boolean result>
207
event Assessment
= f(event+data) result
Other data or event
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
208
Reaction:Assessment:Structure
• EPTS Glossary comparison
– Constraint (also event pattern constraint): A Boolean
condition that must be satisfied by the events observed in
a system.
– Event sink (event consumer) is an entity that receives
events.
• Examples:
• ƒ Software module
• ƒ Database
• ƒ Dashboard
• ƒ Person
208
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
209
Reaction:Assessment:Implementations
• TIBCO, Prova, Oracle, IBM implementations for this
pattern
209
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
event Assessment
= f(event+data) result
Other data or event
epts event processing technical society
210
Reaction:Assessment:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event,
assessment-data
<if>
event-matches-assessment-data
<then>
assign <assessment result>
to <event object>
and/or <other>
• Effect: match event to assessment criteria
and store result 210
event
Assess
event
against
data
event
Store/act on
result
Past event
/ data
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
211
Reaction:Assessment:Implementations: Prova: Rules
• General pattern: <rcvMsg / rcvMult> Event :-
assessment condition(s),
....
assessment rule 1 :-
conditions.
assessment rule 2 :-
filter conditions.
...
• Effect: match event to assessment criteria
and react result
211
event
Assess
event
against
data
event
react on
result
Past event
/ data
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
212
Reaction:Assesment:Implementations: Prova: Example – Detect Suspicious Logins
% detect suspicious logins by assessing the IP numbers
of the login events from the same user login
rcvMsg(XID,Protocol,From,request,login(User,IP)) :-
% if the next follow up event (@count(1)) that follows
the previous login is send from another IP (IP2!=IP)
@group(g1) @count(1)
rcvMsg(XID,Protocol,From,request,login(User,IP2))
[IP2!=IP],
println(["Suspicious login",User,IP,IP2]," ").
212
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
213
Reaction:Assessment:Implementations: Oracle EP: queries
• General pattern:
SELECT <property>)
FROM <SOURCE>,
<ASSESSMENT-RELATION>
WHERE <assessment-condition>
• Effect: join with assessment relation, and
verify assessment conditions.
213
data
join and assess
info
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
event
epts event processing technical society
214 214
Reaction:Assessment:Implementations: IBM WODM Decision Server Events
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
• Example: Assess customer enters branch event
against whether this is a customer of interest
(customer has carried out large banking transactions
recently)
epts event processing technical society
215 215
/* (see Selection:Filtering) */
Reaction:Assessment:Implementations: IBM InfoSphere Streams (SPL)
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
216
RA:Reaction:Routing:
Routing: based on the event type and data pass the event on
to the appropriate service
– E.g. customer purchase event: pass on to a provisioning
service based on the type of product / product classification
216
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
217
Reaction:Routing:Classification
• Alternative Names:
– Event Gateway
• EPTS Reference Architecture:
– "During event reaction, event routing is the process by
which an event is redirected to some process,
computation element, or other event sink. "
• EPIA: no direct analogy, possibly maps to split /
compose / project
– Project (EPA): a translate EPA that takes an input event
and creates a single derived event containing a subset
of the attributes of the input event. 217
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
218
Reaction:Routing:Description
• Role / Explanation
– Routing is used to describe the process of adding one or
more new destinations to an event.
– For example: route an input event to the appropriate
specialist agent for that event type.
• Associated Business Rule Specification
– a constraint (to be enforced during event processing):
The <event> that
satisfies <selection constraint>
must be assigned to
<destination>.
218
event Routing
dest = f(event+data) event
Other data
or event
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
219
Reaction:Routing:Structure
• EPTS Glossary comparison
– Routing (a process on events) is not defined, but is
related to associating an event to an event sink.
– Event sink (event consumer) is an entity that receives
events.
• Examples:
• ƒ Software module
• ƒ Database
• ƒ Dashboard
• ƒ Person
219
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
220
Reaction:Routing:Implementations
• TIBCO, Oracle, Prova and IBM implementations for
this pattern
220
event Routing
dest = f(event+data) event
Other data
or event
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
221
Reaction:Routing:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event,
routing-data
<if>
event-matches-routing-data
<then>
create-new-routed-event and
send-new-routed-event
• Effect: match events to routing rule
and construct / send new event 221
event
Identify
event for
routing
event
Create routed
event
Routing
data
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
222
Reaction:Routing:Implementations: Oracle EP: query pattern
• General pattern template:
Route 1: <filtering pattern> query
Route 2: <filtering pattern> query
Route n: <filtering pattern> query
• Effect: Different predicates
associated to different queries
evaluate and select appropriate
destinations
222
event
Identify
event for
routing
event
Select destination
Routing
data
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
223
Reaction:Routing:Implementations: Oracle EP: model
223
• Scenario:
– Output stocks whose symbols start with ‘a-m’ to
destination 1, and whose symbols start with ‘n-z’ to
destination 2.
• Application Model
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
224
Reaction:Routing:Implementations: Oracle EP: queries
224
Destination-1:
SELECT *
FROM StockTickStream
WHERE symbol.matches(“^[a-m]”)
Destination-2:
SELECT *
FROM StockTickStream
WHERE symbol.matches(“^[n-z]”)
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
225
Reaction:Routing:Implementations: Prova: rules
• General pattern template:
rcvMsg <Event Msg Pattern> :-
<routing decisions>,
sendMsg <route event >.
<routing decision rule 1> :-
<decision conditions>.
<routing decision rule 2> :-
<decision conditions>.
...
• Effect: Events are routed
according to the routing decision
rules 225
event
Identify
event for
routing
event
Create routed
event
Routing
data
Routing
Decisions
Route
event
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
226
Reaction:Routing:Implementations: Prova: Example with Agent (Sub-) Conversations
rcvMsg(XID,esb,From,query-ref,buy(Product) :-
routeTo(Agent,Product), % derive processing agent
% send order to Agent in new subconversation SID2
sendMsg(SID2,esb,Agent,query-ref,order(From, Product)),
% receive confirmation from Agent for Product order
rcvMsg(SID2,esb,Agent,inform-ref,oder(From, Product)).
% route to event processing agent 1 if Product is luxury
routeTo(epa1,Product) :- luxury(Product).
% route to epa 2 if Product is regular
routeTo(epa2,Product) :- regular(Product).
% a Product is luxury if the Product has a value over …
luxury(Product) :- price(Product,Value), Value >= 10000.
% a Product is regular if the Product ha a value below …
regular(Product) :- price(Product,Value), Value < 10000.
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
227
Reaction:Routing:Implementations: Prova: Event Routing in Event-Driven Workflow
rcvMsg(XID,Process,From,event,["A"]) :-
fork_b_c(XID, Process).
fork_b_c(XID, Process) :-
@group(p1) rcvMsg(XID,Process,From,event,["B"]),
execute(Task1), sendMsg(XID,self,0,event,["D"]).
fork_b_c(XID, Process) :-
@group(p1) rcvMsg(XID,Process,From,event,["C"]),
execute(Task2), sendMsg(XID,self,0,event,["E"]).
fork_b_c(XID, Process) :-
% OR reaction group "p1" waits for either of the two
event message handlers "B" or "C" and terminates the
alternative reaction if one arrives
@or(p1) rcvMsg(XID,Process,From,or,_).
227
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
A
B
C
Task1
Task2
D
E
epts event processing technical society
228
Reaction:Routing:Implementations: IBM WODM Decision Server Events
• Example: Define an event rule
which will send the request to supplier A
when the purchase indicates supply
from Supplier A
• Effect of example:
Events are routed as required
228
event
event
Routing
information
Consume
event
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
Identify
event
routing
epts event processing technical society
229
Reaction:Routing:Implementations: IBM CICS Transaction Server for z/OS
• Example: Define a EP adapter for a CICS
Event Binding (i.e. group of related events)
which will route (and format) the event for
delivery to a consumer
• Effect of example: Events routed as required 229
event
Identify
event
routing
event
Routing
information
Consume
event
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
230 230
type Bracket = enum { YOUNG, OLD, OTHER }; /*...*/ stream<rstring name, int32 age> Folks = /*...*/ stream<rstring name, Bracket bracket> Rated = /*...*/ (stream<Rated> Young; stream<Rated> Old) = Custom(Rated) { logic onTuple Rated: { if (bracket == YOUNG) submit(Rated, Young); else if (bracket == OLD) submit(Rated, Old); } }
Reaction:Routing:Implementations: IBM InfoSphere Streams (SPL)
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
• Example:
• Effect: Route based on age rating
epts event processing technical society
231
RA:Reaction:Prediction:
Prediction: an output of some event is the prediction that a
new event will occur (or will have occurred)
– E.g. based on some sequence of sensor events, an
earthquake event is predicted in location L and within time
period P
231
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
232
Reaction:Prediction:Classification
• Alternative Names:
– Event Anticipation; subtype of Event Assessment
• EPTS Reference Architecture:
– Event Prediction is where the reaction to some event
processing is that some new event is predicted to occur.
• Example, a complex event detection may predict some future
event that should then be assessed and processed further.
• EPIA: no direct analogy, possibly maps to split /
compose / project
– Project (EPA): a translate EPA that takes an input event
and creates a single derived event containing a subset
of the attributes of the input event. 232
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
233
Reaction:Prediction:Description
• Role / Explanation
– Prediction is used to describe the process of evaluating the
state of a set of events to predict some future event.
– For example: a new purchase event of an airline ticket may
cause a prediction that a business trip event will occur
• Associated Business Rule Specification
– a constraint (to be enforced during event processing):
The <events> that satisfies <event relationships>
causes a prediction of
<anticipated event>
233
event Prediction
= f(event+data)
predicted
event
Other data or events
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
234
Reaction:Prediction:Structure
• EPTS Glossary comparison
– Virtual event: An event that does not happen in the
physical world but is imagined, modeled or simulated.
• Example: Events predicted by a weather simulation
• Note: A virtual event can refer to either an event object or a
thing that happens.
234
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
235
Reaction:Prediction:Implementations
• TIBCO, IBM, Oracle implementations for this pattern (Prova – Prova Abductive and Inductive Logic Programming or
Java API Call to external functionality)
235
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
event Prediction
= f(event+data)
predicted
event
Other data or events
epts event processing technical society
236
Reaction:Prediction:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event,
prediction-data
<if>
event-matches-prediction-data
<then>
create <predicted event object>
• Effect: match event to prediction criteria
and create predicted event or object
236
event
Assess
event
against
data
event
Compute
prediction
Past event
/ data
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
237
Reaction:Prediction:Implementations: TIBCO BusinessEvents: State Model
• General pattern: <on> event
<condition>
event-matches-prediction-data
<then change state to>
predicted-event-state
• Effect: match event to prediction criteria
and change to predicted state
• Note that there may be a set of prior states that form part
of the state model for predicting events
237
event
Assess
event
against
data
Past event
/ data
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
Prior states
Predicted
state
epts event processing technical society
238
Reaction:Prediction:Implementations: Oracle EP: queries
• Effect: integration with ODM (Oracle
Data Mining) to import model and do
predictive score in an online fashion.
238
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
239 239
Reaction:Prediction:Implementations: IBM WODM Decision Server Events
239
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
• General Pattern: Prediction in WODM is provided
by the Decision Server Rules component – the Rule
Designer API provides classes and interfaces for a
BRL prediction engine
epts event processing technical society
240 240
use com.ibm.streams.mining.scoring::*; type Person = tuple<rstring clientId, int32 age, rstring gender>; type Predict = Person, tuple<float64 predictedVal, float64 predictedStdDev>; /*...*/ stream<Person> People = /*...*/ stream<Predict> Predictions = Regression(People) { param //PMML (predictive model markup language) generated e.g. by SPSS model : "../models/linreg.pmml"; clientId : "CLIENT_ID"; age : "AGE"; gender : "GENDER"; }
Reaction:Prediction:Implementations: IBM InfoSphere Streams (SPL)
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
• Example:
• Effect: Regression operator calculates predicted values and standard
deviation for each tuple in the input stream of People, and assigns to the
output stream (giving predictions about age, gender)
epts event processing technical society
241
RA:Reaction:Discovery:
Discovery: in event processing some new type, schema, state
or classification of events is discovered
– An event pattern instance may be detected, where as an
event pattern class is discovered
– E.g. a customer query event may indicate a combination of
prior transaction events are a new discovered potential fraud
event
241
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
242
Reaction:Discovery:Classification
• Alternative Names:
– Event Mining (c.f. Data Mining)
– subtype of Event Assessment,
parent type of Event Prediction
• EPTS Reference Architecture:
– Event Discovery is where the reaction to some event
processing is the disclosure of a new, typically complex,
event type.
• For example, a complex event detection may be a new event
discovery, or it may simply be detection of an existing event type.
– Note: event prediction is predicting some future event, usually
of a known type, whereas event discovery is the uncovering
of a new event type. See also event-based learning. 242
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
243
Reaction:Discovery:Classification(ctd)
• EPTS Reference Architecture:Note
– Event prediction is predicting some future event, usually
of a known type, whereas event discovery is the
uncovering of a new event type. See also event-based
learning.
• EPIA: no direct analogy, possibly maps to split /
compose / project
– Project (EPA): a translate EPA that takes an input event
and creates a single derived event containing a subset
of the attributes of the input event.
243
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
244
Reaction:Discovery:Description
• Role / Explanation
– Discovery is used to describe the process of evaluating the
state of a set of events to determine some useful information.
– For example: a history of shop purchases may be used to
classify the type of shopper
• Associated Business Rule Specification
– a constraint (to be enforced during event processing):
The <events> that satisfies <event relationships>
causes a discovery of
<related fact>
244
event Discovery
= f(event+data)
discovered
data
Other data or events
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
245
Reaction:Discovery:Structure
• EPTS Glossary comparison
– Event pattern discovery: Finding new event patterns
245
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
246
Reaction:Discovery:Implementations
• TIBCO, Oracle, IBM implementations for this pattern (Prova – Prova Abductive and Inductive Logic Programming or
Java API Call to external functionality)
246
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
event Discovery
= f(event+data)
discovered
data
Other data or events
epts event processing technical society
247
Reaction:Discovery:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event,
past-data
<if>
event-matches-discovery-data
<then>
update <event object>
• Effect: match event to discovery criteria
and add new data to event object
247
event
Assess
event
against
data
event
Compute
discovered
data
Past event
/ data
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
248
Reaction:Discovery:Implementations: Oracle EP: queries
• General pattern:
SELECT <property>
FROM <SOURCE>,
<DISCOVERY-RELATION>
WHERE <discovery-condition>
• Effect: join with discovery relation, and
verify discovery conditions.
248
data
discover
info
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
event
epts event processing technical society
249 249
/* SPL allows you to write your own operator */
Reaction:Discovery:Implementations: IBM InfoSphere Streams (SPL)
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
250
RA:Reaction:Learning:
Learning: uses new event information to refine the knowledge
of event patterns, for example using statistics
– E.g. a neural network for fraud detection uses a learning
mode using historic events versus results in order to train a
new neural net for predicting possible fraud events
250
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
251
Reaction:Learning:Classification
• Alternative Names:
– Event Analytics (c.f. Predictive Analytics)
– Event System Training
– Subtype of Event Assessment
• EPTS Reference Architecture:
– Event learning, or event-based learning, is the reaction to
some event processing that uses new event information
to add to some, typically statistical-based, understanding
of events.
– For example, the use of new event information to provide
closed-loop feedback to adjust event detection or
prediction thresholds. 251
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
252
Reaction:Learning:Classification(ctd)
• EPTS Reference Architecture:Notes
– Event-based learning is a specialisation of general
machine learning and predictive analytics.
– c.f. Event-driven learning is where some (complex) event
determines that some conventional data-driven analytics
requires re-training or re-modelling
• EPIA: no direct analogy, possibly maps to split /
compose / project
– Project (EPA): a translate EPA that takes an input event
and creates a single derived event containing a subset
of the attributes of the input event.
252
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
253
Reaction:Learning:Description
• Role / Explanation
– Learning is used to describe the process of evaluating the state
of a set of events to identify or refine event processes.
– For example: evaluating history of shop purchases may
determine the classification rule criteria for the type of shopper
• Associated Business Rule Specification
– a constraint (to be enforced during event processing):
The <events> that satisfies <event relationships>
causes the learning of
<new event relationship>
253
event Learning
= f(event+data)
New
f(event)
Other data or events
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
254
Reaction:Learning:Structure
• EPTS Glossary comparison
– See Discovery?
254
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
255
Reaction:Learning:Implementations
• TIBCO, IBM implementations for this pattern (Prova – Prova Abductive and Inductive Logic Programming or
Java API Call to external functionality)
255
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
event Learning
= f(event+data)
New
f(event)
Other data or events
epts event processing technical society
256
Reaction:Learning:Implementations: TIBCO BusinessEvents: Rules
• General pattern: <declare> event,
past-data
<if>
event-matches-learning-data
<then>
update <event pattern process>
• Effect: match event to learning criteria
and update some rule condition criteria
256
event
Assess
event
against
data
Manage new
process
Compute
learnt
process
Past event
/ data
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
257 257
/* typically done by using Hadoop from SPL via HDFS adaptors */
Example of HDFS: namespace com.ibm.streams.bigdata.hdfs;
composite SourceExample {
graph
stream<rstring filename> filenames = HDFSDirectoryScan() {
param directory : "test";
}
stream<rstring writer, int32 number> tupleStream
= HDFSFileSource(filenames) {
param
format: txt:
}
// Test reading and the lines of a file
stream<rstring lines> lines = HDFSFileSource(){
param
file : "test/tuple_counter.txt";
format : line ;
}
}
Reaction:Learning:Implementations: IBM InfoSphere Streams (SPL)
Event Reaction Assessment, Routing, Prediction,
Discovery, Learning
epts event processing technical society
258 258
Agenda
• Non-RA EP Patterns
– Other Patterns
– Use Cases using Patterns
epts event processing technical society
259
Other:Application-Time
• Problem:
– Need to use application’s view of time, instead of CPU wall-clock.
– This is particularly useful when events originate from different machines and need some way of synchronizing.
• Scenario:
– Again, consider bid and ask stream…
– However, bid and ask requests are time-stamped on trader at the time that the order is placed.
259
epts event processing technical society
260
Other:Application-Time
• Scenario:
– Seller places two ask requests respectively at time
8:00:00 and 8:00:12
– Buyer places one bid request at time 8:00:11
260
epts event processing technical society
261
Other:Application-Time
• Scenario:
– Remember that we want to correlate using a 10 seconds
window, as anything older is stale…
– Hence first event from seller should not be considered,
and we should correlate the ask price of 11.0 with the bid
price of 9.5
261
epts event processing technical society
262
Other:Application-Time
• Scenario:
– However, the reality is that the first two events could
arrive together in a burst in the exchange, and the third
could be delayed in the cloud…
262
epts event processing technical society
263
Other:Application-Time
• Scenario:
– In this case, bid price of 9.5 would correlate to ask price of 10.0
and the exchange would loose money as the spread is lower…
263
epts event processing technical society
264
Other:Application-Time:Implementations: Oracle EP
• Solution:
– The query does not change…
– However STREAMS must be configured to use
application time-stamps based upon some event
property, instead of having events being system-
timestamped…
264
epts event processing technical society
265
Other:Application-Time:Implementations: IBM Business Monitor
• Example: Define a correlation expression
• Effect of example: The duration of a customer order can be obtained from
the timestamps on start and end events
265
‘Order Complete’
event
Correlate
Calculate
Duration
‘Order Placed’
event
epts event processing technical society
266
Other:Application-Time:Implementations: IBM WODM Decision Server Events
• Example: Events in WODM carry both the time the event
occurred (a timestamp in the event payload) and the time
when the event was received into Decision Server Events
• The example shows event XML received by Decision Server
Events (in one of the supported XML formats) with the
application-time time stamp
<?xml version="1.0" standalone="yes"?>
<connector name="OrderingPatterns" version="2.2">
<connector-bundle name="About_to_place_order" type="Event">
. . .
</connector-bundle>
<system>GBIBMIYACICSAOR1</system>
<timestamp>2012-04-19T15:41:12+00:00</timestamp>
</connector>
266
epts event processing technical society
267
Other:Application-Time:Implementations: Prova: rules
• General pattern template:
rcvMsg <Event Msg Pattern(s)> :-
<add application timestamp to event>,
sendMsg <Adjusted Event>.
• Effect: Any timestamp can be adjusted
(for time computations)
267
Event
adjust event
Updated
Event
epts event processing technical society
268
Other:Application-Time:Implementations: TIBCO BusinessEvents: PreProcessor Pattern
• General Pattern: adjust Event metadata (e.g.
Timestamp) in preprocessor for that event
– events are generally immutable!
– PreProcessor applies regardless of engine used!
• Effect: Any custom timestamp can be adjusted (for time
computations)
268
Event
Preprocess:
adjust event
Updated
Event
Note: If the problem is “out of order” events
(rather than synchronizing clocks) a
different pattern may be better!
epts event processing technical society
269 269
Agenda
• EP Patterns in Use Cases
epts event processing technical society
270
ATM/Branch
IBM Business
Monitor
SMS System
Transactions
Decision Server Events
CICS TS
Actions
Internet
Banking
Customer
Profile
WebSphere MQ
Customer Deposit and
Withdrawal
Transactions from CICS
1
Process Customer
Deposit and
Withdrawal Events
and identify event
patterns
2
Determine
appropriate
customer
promotion
4
Customer/RM SMS
5
Collect
Customer’s
profile
3 Decision
Server Rules
Measure success of
promotions 6
WebSphere
Operational
Decision
Management
IBM CICS and WebSphere Operational Decision Management Use Case
epts event processing technical society
271
Oracle Use-case: Capital Markets
• Low-Latency Trading
– Algorithm Trading
• Trade Matching
– High throughput
– In-flight fraud
• Position Capture and Aggregation
• Risk and Analytics
– Pre/post trade analytics
– On-demand risk management
271
epts event processing technical society
272
Oracle Use-case: Telemetry and Geo-fencing
• Spatial extension in CQL
– Geometry objects (point, polygon)
– Operations: within, inside, touches,
etc.
• Allows queries such as:
– Is streaming location of vehicle within
pre-defined areas?
– Is streaming customer close to shop?
272
epts event processing technical society
273
TIBCO Use Case: Application Service Gateway Telco high performance service policies using event processing
273
Mobile Services
difficult to monitor
end to end
Auto detect and
fixing of issues
for better
user service
Excellent service quality
needed to avoid
customer churn
Service quality affects
take-up of new services
epts event processing technical society
274
• Standardized channel for monitoring subscribers
• Standardized access to services (e.g. Facebook, Google, etc)
• Consolidation of existing multiple gateways
• Allows analysis of service usage, subscriber behavior, service performance, marketing campaign success, etc
• Creates events for revenue generation and reconciliation
• Handles Common Telco Notifications - change-of-device, contract, service- allowing UpSell/Cross sell etc
Service Gateway architecture exploits event processing
epts event processing technical society
275
• Standardized channel for monitoring subscribers
• Standardized access to services (e.g. Facebook, Google, etc)
• Consolidation of existing multiple gateways
• Allows analysis of service usage, subscriber behavior, service performance, marketing campaign success, etc
• Creates events for revenue generation and reconciliation
• Handles Common Telco Notifications - change-of-device, contract, service- allowing UpSell/Cross sell etc
Service Gateway architecture exploits event processing
Routing Pattern
Transformation
Patterns
Decision / Policy
Pattern
Security Pattern
epts event processing technical society
276
TIBCO Use Case: Service Performance Manager SOA metrics generation using event processing
276 TIBCO BW Domain
ESB
Java
Matrix
.NET
BW
Matrix
Instrumentation & Management Bus (EMS/JMX)
TIBCO ActiveMatrix Enterprise
Alert &
Assure SLA
Analyze
TIBCO ActiveMatrix SPM Monitoring Dashboard
SLA Dashboard
Rules
Dash
boa
rd M
essa
ge B
us
epts event processing technical society
277 277
Agenda
• EP Pattern Classifications
• EP Patterns @EPIA
epts event processing technical society
278
Categorization of Event Processing Patterns
Source:Paschke, A.: Design Patterns for Complex Event Processing, DEBS'08, Rome, Italy, 2008
http://arxiv.org/abs/0806.1100v1
epts event processing technical society
279
Categorization of Patterns
• Categorization according to Good and Bad Solutions – CEP Patterns
– CEP Anti-Patterns
• Categorization according to the Abstraction Level – Guidelines and Best Practices
– Management patterns
– Architecture patterns
– Design patterns
– Mapping patterns
– Idioms / Realization patterns
– Smells / Refactoring patterns
• Categorization according to the Intended Goal – Adoption patterns
– Business patterns
– Integration patterns
– Composite patterns:
– …
• Categorization according to the Management Level – Strategic patterns
– Tactical patterns
– Operational patterns Source:Paschke, A.: A Semantic Design Pattern Language for Complex Event Processing,
Intelligent Event Processing, Papers from the 2009 AAAI Spring Symposium (2009) , p. 54-60.
http://www.aaai.org/Papers/Symposia/Spring/2009/SS-09-05/SS09-05-010.pdf
epts event processing technical society
280
Pattern categorization from ‘Event Processing in Action’
1. Filtering
– stateless or stateful
2. Transformation
– Project
– Translate
– Enrich
– Split
– Aggregate
– Compose
3. Pattern Detection - the types of pattern that can be detected
– Basic patterns
• Logical Operator patterns
– ALL
– ANY
– ABSENCE
– Threshold patterns
– COUNT
– VALUE MAX
– VALUE MIN
– VALUE AVERAGE
– FUNCTOR
• Subset selection patterns
– RELATIVE n HIGHEST VALUES
– RELATIVE n LOWEST VALUES
– Modal patterns
• ALWAYS
• SOMETIMES
– Dimensional patterns
• Temporal order patterns
– SEQUENCE
– FIRST n
– LAST n
• Temporal Trend patterns
– INCREASING
– DECREASING
– STABLE
– NON-INCREASING
– NON-DECREASING
– MIXED
• Spatial patterns
– MAX DISTANCE
– AVERAGE DISTANCE
– RELATIVE MIN DISTANCE
– RELATIVE MAX DISTANCE
– RELATIVE AVERAGE DISTANCE
• Spatiotemporal patterns
– MOVING IN A CONSTANT DIRECTION
– MOVING IN A MIXED DIRECTION
– STATIONARY
– MOVING TOWARD
epts event processing technical society
Summary
• Functional Patterns v1.0 are
“complete”
• Next: further analysis and refinement
terminology
+ more examples
+ drill-down into operation level
281
epts event processing technical society
282
References
• EPTS Glossary
http://www.ep-ts.com/component/option,com_docman/task,doc_download/gid,66/Itemid,84/
• Event Processing in Action
– Opher Etzion and Peter Niblett, Manning Publications Co. (2011)
– http://www.manning.com/etzion/