detecting changes in streaming data using adaptive...
TRANSCRIPT
DetectingchangesinstreamingdatausingadaptiveestimationDeanBodenham,ETHZürich(jointworkwithNiallAdams)
Loading [MathJax]/jax/output/HTML-CSS/jax.js
OutlineChangedetectioninstreamingdataApplicationsAdaptiveestimationusingforgettingfactorsExtensions
Streamingdata
DatastreamsAdatastreamisasequenceofrandomobservations
observedsequentiallypotentiallyunendingtoomanyobservationstostoreandanalyselater
observationsarriveatahighfrequencyunderlyingdistributionsareoftenunknownunpredictablewhenandhowchangesoccur
, ,… , ,…x1 x2 xN
Changedetection
Changedetectioninstreamingdata
Changedetectioninstreamingdata
Changedetectioninstreamingdata
Changedetectioninstreamingdata
Changedetectioninstreamingdata
Changedetectioninstreamingdata
Changedetectioninstreamingdatasequentiallydetect(significant)changesinthestreamdetectthefirstchangepointandthencontinuouslymonitorforfurtherchangepoints
useasfewcontrolparametersaspossible
Changedetectionusingacontrolchart
ChangedetectionusingacontrolchartAcontrolchartconsistsof:statisticscontrollimits ,whereisachangepointif:
for
Wesay:in-controlout-of-control
, ,…z1 z2a, b a < b
τ∈ (a, b)zi i = 1,2,… , τ
∉ (a, b)zτ+1
∈ (a, b) ⇒zi
∉ (a, b) ⇒zi
Changedetectionusingacontrolchart
ChangedetectioninstreamingdataWeassume:observationsrealisationsofrandomvariablesdistributions
Wesaythat arethechangepointsofthedatastream.
, ,…x1 x2, ,…X1 X2
≠ ≠ …D1 D2
, ,… ,X1 X2 Xτ1
, ,… ,X +1τ1 X +2τ1 Xτ2
, ,… ,X +1τ2 X +2τ2 Xτ3
∼ ,D1
∼ ,D2
∼ ,etcD3
, ,…τ1 τ2
Changedetection:examples
Changedetection:examples
Changedetection:literatureOriginalmethod:Shewhartcontrolcharts: ,forparameter
Twopopularmethods:CUSUM:CumulativeSumEWMA:ExponentiallyWeightedMovingAverageBothneedtwocontrolparametersIfthedistributionsareknown,CUSUMhasoptimalityproperties
ParameterchoicesdependonmagnitudeofthechangeModernmethodsusuallyrequiretwoormorecontrolparametersManymethodsonlyfocusonasinglechange
( + Ks, − Ks)x̄ x̄ K
Changedetectioninstreamingdata
sequentiallydetectmultiplechangepointsinastream
Changedetectioninstreamingdata
sequentiallydetectmultiplechangepointsinastream
Changedetectioninstreamingdata
sequentiallydetectmultiplechangepointsinastream
Changedetectioninstreamingdata
sequentiallydetectmultiplechangepointsinastream
Changedetectioninstreamingdata
sequentiallydetectmultiplechangepointsinastream
Changedetectioninstreamingdata
sequentiallydetectmultiplechangepointsinastream
Changedetectioninstreamingdata
sequentiallydetectmultiplechangepointsinastream
Changedetectioninstreamingdata
sequentiallydetectmultiplechangepointsinastream
DifferencetoofflinemethodsTraditionalmethods Streamingdatamethods
fixedamountofobservations unending
accesstoallofthedataallthetime onepass
(often)prechangedistribution isknown unknown
(often)onlymonitoringforasinglechangepoint multiplechangepoints
Lastpointrelatedtoqualitycontrol:canwestoptheprocess?Streamingdataisoftenjustobserved,cannotbestopped
F1
ContinuousmonitoringMonitorapotentiallyunendingstreamExpectmutiplechangepointsAfterfirstchangepoint,monitorfornextchangepointsUseasfewcontrolparametersaspossible
ApplicationsCybersecurityFinanceBiologyandmedicine
Adaptiveestimationwithforgettingfactors
SamplemeanStream (mostrecent: )
Allobservationshaveequalweight
, ,… ,x1 x2 xN xN
x̄N
x̄N
= [ + + ⋯ + + ]1N
xN xN−1 x2 x1
= ,1N
∑i=1
N
xi
Samplemean:sequentialFor
Settingfor‘mass’and for‘weight’
i = 1,2,…mi
wi
x̄i
= +mi−1 xi
= + 1wi−1
=mi
wi= = 0m0 w0
m w
ForgettingfactormeanBuildingonworkofChristoforosAnagnostopoulosandNiallAdams,
Stream: (mostrecent: )Usingaforgettingfactor
Earlierobservationsdownweighted
, ,… ,x1 x2 xN xN
λ ∈ [0, 1]
x̄N,λ
x̄N,λ
= [ + λ + … + ]1
wN,λxN xN−1 λN−2x2 λN−1x1
= , =1
wN,λ∑i=1
N
λN−ixi wN,λ ∑i=1
N
λN−i
Forgettingfactormean:sequentialFor
Setting
i = 1,2,…mi,λ
wi,λ
x̄i,λ
= λ +mi−1,λ xi
= λ + 1wi−1,λ
=mi,λ
wi,λ= = 0m0,λ w0,λ
Aside:Twospecialcases
But::onlylatestobservation:samplemean
Usuallyonlyconsider
=x̄N,λ1
wN,λ∑i=1
N
λN−ixi
λ ∈ [0, 1]
λ = 0λ = 1
λ ∈ (0, 1)
Asimulationstudy
Changefrom toN(0, 1) N(3, 1)
Asimulationstudy
Averagebehaviourover100trials
Valueof ?Problem:Howdowechoose ?
Onesolution:Adaptiveforgettingfactor
λλ
λ→
Adaptiveforgettingfactors
AdaptiveforgettingfactormeanGivenasequenceofobservations ,andforgettingfactors ,
Theadaptiveforgettingfactormean isdefined,for ,by
, ,… ,x1 x2 xN
, ,… ,λ1 λ2 λN−1m
N, λ→
wN, λ
→
= + ,λN−1mN−1, λ
→ xN
= + 1,λN−1wN−1, λ
→
= 0,m0, λ
→
= 0,w0, λ
→
x̄N, λ
→ N ≥ 1
=x̄N, λ
→
mN, λ
→
wN, λ
→
UpdatingUsegradientdescent:
→ ?λN λN+1
= − η , η ≪ 1λN λN−1∂
∂ λ→ L
N, λ→
DerivativeofForanyfunction involving :
λ→
fN, λ
→ λ→
∂
∂ λ→ f
N, λ→ = [ − ]lim
ϵ→0
1ϵ
fN, +ϵλ
→ fN, λ
→
Example:m
N, λ→
mN, λ
→
mN, +ϵλ
→
= +λN−1mN−1, λ
→ xN
= [( ) ]∑i=1
N ∏p=i
N−1
λp xi
= [( ( + ϵ)) ]∑i=1
N ∏p=i
N−1
λp xi
(1)
(2)
(3)
Example:Δ
N, λ→
ΔN, λ
→
ΔN, λ
→
= = [ − ]∂
∂ λ→ m
N, λ→ lim
ϵ→0
1ϵ
mN, +ϵλ
→ mN, λ
→
= [ ( ) ]∑i=1
N−1 ∑t=i
N−1 ∏p=ip≠t
N−1
λp xi
= + , = 0λN−1ΔN−1, λ
→ mN−1, λ
→ Δ1, λ
→
(4)
(5)
(6)
ChoiceofcostfunctionThefollowingcostfunctionisused:
Inthegradientdescentstep:
LN, λ
→ = [ −x̄N−1, λ
→ xN ]2
λN = − ηλN−1∂
∂ λ→ L
N, λ→
Behaviourofadaptiveforgetting
Asimulationstudy
Averagebehaviourover100trials
Asimulationstudy:withAFF
Averagebehaviourover100trials
Decidingwhenachangehasoccurred
Expectationandvarianceof
Assume:datastreamissampledfromthei.i.d. ,with
Then:
where and,for ,
X̄N, λ
→
, ,… ,X1 X2 XN
E[ ] = μXi
Var[ ] =Xi σ2
E[ ]X̄N, λ
→
Var[ ]X̄N, λ
→
= μ,
= ( ) ,uN, λ
→ σ2
= 1u1, λ
→ i ≥ 1
= (1 − + ( .ui+1, λ
→1
wi+1, λ
→)2
ui, λ
→1
wi+1, λ
→)2
AdecisionruleAssumingnormalityUsenormalcdftofindcontrollimits(a,b)
Then:nochange
change
Pr[ ≤ a] = α/2, Pr[ ≥ b] = 1 − α/2,X̄N,λ X̄N,λ
∈ (a, b) ⇒X̄N,λ∉ (a, b) ⇒X̄N,λ
ChangedetectionwithAFFmethodAdaptiveforgettingfactor(AFF)methodonecontrolparametercomputedsequentiallyfixedmemoryandcomputationalrequirements(fornow)assumesnormalityfordecisionrule
α
Simulationstudy
PerformancemeasuresStandard:ARL1:Howquicklyisatruechangeisdetected?ARL0:Howlonguntilafalsechangeisdetected?
Forcontinuousmonitoring:DNF:ProportionofdetectionsnotfalseCCD:Proportionofdetectionscorrectlydetected
Simulationstudy
Algo Parameters Values CCD DNF ARL1 ARL0
CUSUM ( , ) (0.50,4.77) 0.85 0.77 24.73 373.38
EWMA ( , ) (0.10,2.814) 0.87 0.78 24.17 420.15
AFF 0.005 0.86 0.79 27.12 819.36
Performanceonastreamofover750,000observationswithapproximately changesofsize ,with
CCD:higherisbetterDNF:higherisbetterARL0:higherisbetterARL1:lowerisbetter
k h
r L
α
5000 δδ ∈ {0.25, 0.5, 1, 3}
Application
Application:ForeignexchangedataGBP/CHFapprox.4yearsofmonitoringevery5minutes330,000observations
Wherearethetruechangepoints?noideacomparedetectedchangepointswithoptimalofflinedetector
ComparingAFFandofflinedetector
Extensions
NaturalextensionstoothersettingsMultivariatecasemonitoreachstream,combinep-values
Changeinthevariance
ExtremevaluedistributionsNonparametricextensions
=s2N,λ
1vN,λ
∑i=1
N
λN−i [ − ]xi x̄N,λ2 (7)
RpackageffstreamavailableonCRAN
Thankyou!
Slidespreparedwith:
rmarkdownreveal.jspdf2svg
References
—andN.M.Adams(2016)Continuousmonitoringforchangepointsindatastreamsusingadaptiveestimation,StatisticsandComputing,—andN.M.Adams(2013)ContinuousmonitoringofacomputernetworkusingmultivariateadaptiveestimationICDMWorkshopsC.AnagnostopoulosC(2010)Astatisticalframeworkforstreamingdataanalysis.PhDthesis,ImperialCollegeLondonS.Haykin(2002)AdaptiveFilterTheory,Prentice-HallR.Killick,P.FearnheadandI.A.Eckley(2012)Optimaldetectionofchangepointswithalinearcomputationalcost,JASAE.Page(1954)Continuousinspectionschemes,BiometrikaS.W.Roberts(1959)Controlcharttestsbasedongeometricmovingaverages,Technometrics
doi:10.1007/s11222-016-9684-8
Extra
Behaviourofadaptiveforgetting
PoemRosesarered,violetsareblue.
Othercolours
chosencolourisdodgerblue,chosencolourisdodgerblue,
Analignλ =
1ϵ + δ
=1ξ
(8)
Anequationwithnumbersλ =
19
(9)