anomaly detection by mean and standard deviation (lt at aq)
DESCRIPTION
TRANSCRIPT
![Page 1: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/1.jpg)
Anomaly Detection
iwanaga
![Page 2: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/2.jpg)
Who am I
@quake_alert@quake_alert_en@quake_alert_fr@quake_alert_kr
Yoshihiro Iwanaga
![Page 3: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/3.jpg)
Motivation for detecting anomaly
Traditional system monitoring
• process existence• ping, http, tcp response• disk usage
→ “fixed” rule / threshold
![Page 4: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/4.jpg)
Motivation for detecting anomaly
Notice something out of ordinary
• network traffic is heavier than usual • number of login try is obviously larger• a colleague is strangely gracious today
→ Unusual behaviors; Indications of fault.
Such info helpspreventing service degrading in advance!!
but rule/threshold vary with service, host, client, time…
![Page 5: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/5.jpg)
key to detect anomaly
usual unusual
Watch differences b/w
![Page 6: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/6.jpg)
e.g. Network Traffic
Mon Tue Wed Thu Fri traffic
time
![Page 7: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/7.jpg)
Superimpose 24 hour plot
Traffic at 15:00 on workdayis about 1.2 Gbps
traffic
timePeriodicity!!
![Page 8: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/8.jpg)
mean
mean - 3σ
mean + 3σ
amount of dispersion from mean
Acceptable “range”
→ e.g. Acceptable range of traffic at 15:00 on workday is1.01 to 1.38 Gbps
![Page 9: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/9.jpg)
Case examples
![Page 10: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/10.jpg)
DDoS
partialhardware failure
Traffic
![Page 11: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/11.jpg)
number of mail passed spam filterspam rate
Applied a wrong spam rule
![Page 12: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/12.jpg)
However
Reality is not that simple…
人生楽ありゃ苦もあるさ涙の後には虹も出る
歩いてゆくんだしっかりと自分の道をふみしめて
山上路夫
![Page 13: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/13.jpg)
downloading large files
mass e-mail sending
“Traffic spike” happens so frequently
Frequent false-positive alerting will be
“cry-wolf” system…
![Page 14: Anomaly Detection by Mean and Standard Deviation (LT at AQ)](https://reader033.vdocuments.us/reader033/viewer/2022061120/546c84f2af79592c498b5e29/html5/thumbnails/14.jpg)
heuristic filtering
In usual, traffic gets cool downwithin 15 minutes
notify engineersif anomaly continues more than 15 minutes
Engineers’ knowledge is gold minefor better algorithm
→ one practical example: