EVENT MANAGEMENT IN MULTIVARIATE STREAMING SENSOR DATA
National and Kapodistrian University of Athens
What is an event?• The term “event” is used to describe an alteration on one
or more variables monitored by the system• Two kinds of processing modules with respect to an event
• Online event processing: focuses on real event detection, identification of time dependent correlations and causalities
• Offline event processing: event storage, post-processing of stored events and data -warehousing
Online event processing
0.8
0.6
0.5
Event
dete
ctio
n
Sensor streams
e1
e2
e3
e1e2
e2e3
E1 • 0• 1• 0• 0
E2 • 0• 1• 0• 0
En • 0• 0• 1• 0
0.2 150.0 0.12
250.0 8.0 248.3
23.0 21.4 342.1
0.15
251.0
22.9
s1
s2
s3
Event streams
Probabilistic Temporal Reasoning
Dependency structure
tt
e1 e2 e3
2 3 1e e e
Event correlation
Event
pre
dic
tion
1 2e e
2 3e eAdaptive
Filtering
Event/Change Detection• Sensor streams arrives as raw data that provide instant
measurements• Generation of event streams over an existing set of
sensor streams• The problem concerns both detecting whether or not a
change has occurred, or whether several changes might have occurred, and identifying the times of any such changes.
Event/Change detection algorithms• Change detection algorithms
• Cumulative Sum (CUSUM)• Shewhart Controller• Multivariate Autoregressive Model (MAR)
CUSUM(1/3)• The input parameters for the CUSUM algorithm are the
following:• the target value μ• the above-tolerance value • the below-tolerance value • the above-threshold value • the below-threshold value
• The output parameters for the CUSUM algorithm are the following:• the above-detection signal {0,1}• the below-detection signal {0,1}
CUSUM (3/3)• Experiment set up
• μ = 0.5• = = 1.3
Uni
vari
ateti
me
seri
es x
t
(Acc
eler
ation
m/s
ec)
Time steps t
Positive change
Negative change
0.5, 0.3, 1.3k k thresh thresh
Time steps t
Cum
ulati
ve s
ums
Positive sum P
Negative sum N
0.5, 0.3, 1.3k k thresh thresh
Shewhart Controller (1/3)• In the Shewhart control chart, a variable is detected to
deviate at time t from its normality whenever exceeds one of the control limits
• Control limits• Upper Control Limit (UCL)
+
• Lower Control Limit (LCL)• = -
Shewhart Controller (3/3)
Univ
aria
tetim
e se
ries
x t(A
ccel
erati
on m
/sec
)
Time steps t
UCL and LCL
Detected change
3k
UCL
LCL
Multivariate Autoregressive (MAR)
Time steps t
Time steps t
Varia
ble
2 es
timati
on
(Lum
inan
cecd
/m2)
1,tx 1,tx
2,tx 2,tx 2
2
Varia
ble
1 es
timati
on
(Lum
inan
cecd
/m2)
Relati
ve Err
or e 2
t
Time steps t
2,te 2thresh Detected change 2 7%thresh
Relati
ve Err
or e 1
t
Time steps t
1,te 1thresh Detected change 1 5%thresh
Event Correlation• Technique for making sense of a large number of events
and pinpointing the few events that are really important in that mass of information
• Accomplished by looking for and analyzing relationships between events.
• Implemented by a piece of software called “event correlator”
Event correlation: step-by-step• Event filtering
• consists in discarding events that are deemed to be irrelevant by the event correlator
• Event aggregation• a technique where multiple events that are very similar (but not
necessarily identical) are combined into an aggregate that represents the underlying event data
• Event masking• consists in ignoring events pertaining to systems that are
downstream of a failed system
• Root cause analysis• It consists in analyzing dependencies between events, based for
instance on a model of the environment and dependency graphs, to detect whether some events can be explained by others
Event Correlation Engine (ECE)• Typical event correlation scheme (univariate data)
• A transition from object (i.e., event or sequence of events) A to object B occurs if and only if B occurs immediately after A (i.e., not within a time window).
• Only one object is considered at each step of the sequence (i.e., there are no objects occurring at the same time).
• Event correlation over multivariate sensor data• an alerting situation or a malfunctioning system is expected to lead
to several events triggered at the same time step.
Correlation of Multivariate Event Data• Stepwise correlation
• Based on a first order Markov chain
• Variable-order correlation of Multivariate Event Data• Based on idea of partial matching [Fan et al. 1999]
• Event correlation based on sliding window• Hybrid scheme that correlates events within a time window
Stepwise Correlation
1 1,0,1EV AC
3 0,0,0EV
= 1ACPA B C
1 0 1
0 1 1
0 0 0
1 0 1
0 1 0
0 1 0
0 0 0
1 0 0
0 1 0
BCP1
=3
BC
2 0,1,1EV
AC
ACP1
=3
AC,BCP = 1
P
1=3
BC,P = 1
BCP1
=2
BCAC
ACP1
=2
AC,BCP = 1
4 1,0,1EV BCP1
=4
BCAC
ACP2
=4
AC,BCP = 1
P
1=4
BC,P = 1
5 0,1,0EV BCP1
=5
BCAC
ACP2
=5
AC,BCP1
=2
P
1=5
BC,P = 1
BAC,BP
1=2
BP1
=5
6 0,1,0EV BCP1
=6
BCAC
ACP2
=6
AC,BCP1
=2
P
1=6
BC,P = 1
BAC,BP
1=2
BP2
=6
,BCP = 1
B,BP = 1
,BCP = 1
,BCP = 1
A B C
1 0 1
0 1 1
0 0 0
1 0 1
0 1 0
0 1 0
0 0 0
1 0 0
0 1 0
7 0,0,0EV BCP1
=7
BCAC
ACP2
=7
AC,BCP1
=2
P2
=7
BC,P = 1
BAC,BP
1=2
BP2
=7
B,BP1
=2
,BCP = 1B,P
1=2
8 1,0,0EV BCP1
=8
BCAC
ACP2
=8
AC,BCP1
=2
P2
=8
BC,P = 1
BAC,BP
1=2
BP2
=8
B,BP1
=2
,BCP
1=2B,P
1=2
AAP
1=8
,AP
1=2
9 0,1,0EV
BCP1
=9
BCAC
ACP2
=9
AC,BCP1
=2
P2
=9
BC,P = 1
BAC,BP
1=2
BP3
=9
B,BP1
=2
,BCP
1=2B,P
1=2
AAP
1=9
,AP1
=2
A,BP = 1
Variable-order correlation • Partial matching algorithm [Fan et al.199]
A B C
1 0 1
0 1 1
0 0 0
1 0 0
0 1 0
A/1 C/1
A/1
C/2 B/1 C/1 BC/1
B/1 C/1 BC/1
AC/1
AC/1
B/1 C/1 BC/1
B/1 BC/1
A/1
B/1 C/1 BC/1 B/1 C/1 BC/1B/1 C/1 BC/1
B/1 BC/1
1 1,0,1EV
/1
2, 1m l
AC/1C/2
2 0,1,1EV
3 0,0,0EV
/1
/1/1/1/1/1/1 /1 /1/1
/1
/1
Variable-order correlation
2, 1m l
A B C
1 0 1
0 1 1
0 0 0
1 0 0
0 1 0
A/2
B/1 C/1 BC/1 B/1 C/1 BC/1B/1 C/1 BC/1
B/1 BC/1
/1 AC/1C/2 /1
/1/1/1/1/1/1 /1 /1/1
/1A/1 A/1
/1
A/1
A/1
4 1,0,0EV
5 0,1,0EV A/2
B/2 C/1 BC/1
B/1 C/1 BC/1B/1 C/1 BC/1
B/2 BC/1
/1
AC/1C/2
/1
/1/1/1
/1/1/1
/1 /1/1
/1
A/1 A/1
/1
A/1
A/1
B/1
Sliding window algorithm• Address time dependencies among events within a specific
timeframe• At each the algorithm the algorithm recalculates probability
values with respect to a sliding window taking into account the new event vector arrived at the current time step t
• The algorithm has memory of exactly w time steps• Directed graph G=(V, E) where V=P(I) is the power set of I={}
• Graph Vertexes :• Weighted transition edge:
Sliding window algorithm• Frequency of each vertex, a – indicator
• For estimating the probabilities within two nodes, b - indicator• The b-indicator examines whether the event sets of two nodes
occur at two, possibly separate, time steps .
Sliding window algorithm• Two steps • First step: t < w
• Frequency of each vertex-node• Probability of occurrence • Frequency of vV within the occurrence of some node u V
• Conditional probability
Sliding window algorithm• Second step: t > w
• Frequency of each vertex-node within the last w time
• Probability of occurrence • Frequency of vV within the last w after the occurrence of some
node u V
• Conditional probability:
Sliding window algorithm
A B C
1 0 1
0 1 0
1 0 0
1 1 0
3w 1 1,0,1EV
AC
CA
1,wAP = 1 1,w
CP = 11,wC,AP = 1
1,wA,CP = 1
1,wC,ACP = 1
1,wAC,CP = 1
1,wA,ACP = 1
1,wAC,AP = 1
2 0,1,0EV
AC
CA
2,wAP
1=2
2,wCP
1=2
2,wC,AP
1=2
2,wA,CP
1=2
2,wC,ACP
1=2
2,wAC,CP
1=2
2,wA,ACP
1=2
2,wAC,AP
1=2
B2,wC,BP
1=2
2,wBP
1=2
1,wACP = 1 2,w
ACP1
=2
2,wA,BP
1=2
2,wAC,BP
1=2
3 1,0,0EV
AC
CA
3,wAP
2=3
3,wCP
1=3
3,wC,AP
2=3
3,wA,CP
1=4
3,wC,ACP
1=33,w
AC,CP1
=3
3,wA,ACP
1=4
3,wAC,AP
2=3
B3,wC,BP
1=3
3,wBP
1=3
3,wACP
1=3
3,wA,BP
1=4
3,wAC,BP
1=3
3,wB,AP
1=2
3,wA,AP
1=2
4 1,1,0EV
A
4,wAP
1=3
B
4,wBP
2=3
4,wA,BP
1=2
4,wB,AP
1=4
4,wA,AP = 1
4,wB,BP
1=2
AB4,wB,ABP
1=2
4,wAB,BP = 1
4,wA,ABP = 1
4,wAB,AP = 1
4,wABP
1=3
Event processing• A method of tracking and analyzing (processing) streams
of information (data) about things that happen (events),
and deriving a conclusion from them• Complex event processing, or CEP, is event processing
that combines data from multiple sources to infer events or patterns that suggest more complicated circumstances
• Techniques for CEP• Event-pattern detection• Event abstraction• Event filtering• Event aggregation and transformation• Modeling event hierarchies
CEP categories
• Two main categories• Aggregation-oriented CEP: an aggregation-oriented CEP solution
is focused on executing on-line algorithms as a response to event data entering the system. A simple example is to continuously calculate an average based on data in the inbound events
• Detection-oriented CEP: focused on detecting combinations of events called events patterns or situations. A simple example of detecting a situation is to look for a specific sequence of events.