qcon rio 2015 - proatividade na análise de logs com elk
TRANSCRIPT
![Page 1: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/1.jpg)
Proatividade na análise de logs com
Elasticsearch, Logstash e Kibana
![Page 2: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/2.jpg)
Olá!Leonardo Comelli | @leocomelli
![Page 3: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/3.jpg)
![Page 4: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/4.jpg)
64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 1284664.242.88.10 - - [07/Mar/2004:16:06:51 -0800] "GET /twiki/bin/rdiff/TWiki/NewUserTemplate?rev1=1.3&rev2=1.2 HTTP/1.1" 200 452364.242.88.10 - - [07/Mar/2004:16:10:02 -0800] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 629164.242.88.10 - - [07/Mar/2004:16:11:58 -0800] "GET /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 200 735264.242.88.10 - - [07/Mar/2004:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 525364.242.88.10 - - [07/Mar/2004:16:23:12 -0800] "GET /twiki/bin/oops/TWiki/AppendixFileSystem?template=oopsmore¶m1=1.12¶m2=1.12 HTTP/1.1" 200 1138264.242.88.10 - - [07/Mar/2004:16:24:16 -0800] "GET /twiki/bin/view/Main/PeterThoeny HTTP/1.1" 200 492464.242.88.10 - - [07/Mar/2004:16:29:16 -0800] "GET /twiki/bin/edit/Main/Header_checks?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 1285164.242.88.10 - - [07/Mar/2004:16:30:29 -0800] "GET /twiki/bin/attach/Main/OfficeLocations HTTP/1.1" 401 1285164.242.88.10 - - [07/Mar/2004:16:31:48 -0800] "GET /twiki/bin/view/TWiki/WebTopicEditTemplate HTTP/1.1" 200 373264.242.88.10 - - [07/Mar/2004:16:32:50 -0800] "GET /twiki/bin/view/Main/WebChanges HTTP/1.1" 200 4052064.242.88.10 - - [07/Mar/2004:16:33:53 -0800] "GET /twiki/bin/edit/Main/Smtpd_etrn_restrictions?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 1285164.242.88.10 - - [07/Mar/2004:16:35:19 -0800] "GET /mailman/listinfo/business HTTP/1.1" 200 637964.242.88.10 - - [07/Mar/2004:16:36:22 -0800] "GET /twiki/bin/rdiff/Main/WebIndex?rev1=1.2&rev2=1.1 HTTP/1.1" 200 4637364.242.88.10 - - [07/Mar/2004:16:37:27 -0800] "GET /twiki/bin/view/TWiki/DontNotify HTTP/1.1" 200 414064.242.88.10 - - [07/Mar/2004:16:39:24 -0800] "GET /twiki/bin/view/Main/TokyoOffice HTTP/1.1" 200 385364.242.88.10 - - [07/Mar/2004:16:43:54 -0800] "GET /twiki/bin/view/Main/MikeMannix HTTP/1.1" 200 368664.242.88.10 - - [07/Mar/2004:16:45:56 -0800] "GET /twiki/bin/attach/Main/PostfixCommands HTTP/1.1" 401 1284664.242.88.10 - - [07/Mar/2004:16:47:12 -0800] "GET /robots.txt HTTP/1.1" 200 6864.242.88.10 - - [07/Mar/2004:16:47:46 -0800] "GET /twiki/bin/rdiff/Know/ReadmeFirst?rev1=1.5&rev2=1.4 HTTP/1.1" 200 572464.242.88.10 - - [07/Mar/2004:16:49:04 -0800] "GET /twiki/bin/view/Main/TWikiGroups?rev=1.2 HTTP/1.1" 200 516264.242.88.10 - - [07/Mar/2004:16:50:54 -0800] "GET /twiki/bin/rdiff/Main/ConfigurationVariables HTTP/1.1" 200 5967964.242.88.10 - - [07/Mar/2004:16:52:35 -0800] "GET /twiki/bin/edit/Main/Flush_service_name?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 1285164.242.88.10 - - [07/Mar/2004:16:53:46 -0800] "GET /twiki/bin/rdiff/TWiki/TWikiRegistration HTTP/1.1" 200 3439564.242.88.10 - - [07/Mar/2004:16:54:55 -0800] "GET /twiki/bin/rdiff/Main/NicholasLee HTTP/1.1" 200 723564.242.88.10 - - [07/Mar/2004:16:56:39 -0800] "GET /twiki/bin/view/Sandbox/WebHome?rev=1.6 HTTP/1.1" 200 854564.242.88.10 - - [07/Mar/2004:16:58:54 -0800] "GET /mailman/listinfo/administration HTTP/1.1" 200 6459lordgun.org - - [07/Mar/2004:17:01:53 -0800] "GET /razor.html HTTP/1.1" 200 286964.242.88.10 - - [07/Mar/2004:17:09:01 -0800] "GET /twiki/bin/search/Main/SearchResult?scope=text®ex=on&search=Joris%20*Benschop[^A-Za-z] HTTP/1.1" 200 428464.242.88.10 - - [07/Mar/2004:17:10:20 -0800] "GET /twiki/bin/oops/TWiki/TextFormattingRules?template=oopsmore¶m1=1.37¶m2=1.37 HTTP/1.1" 200 1140064.242.88.10 - - [07/Mar/2004:17:13:50 -0800] "GET /twiki/bin/edit/TWiki/DefaultPlugin?t=1078688936 HTTP/1.1" 401 1284664.242.88.10 - - [07/Mar/2004:17:16:00 -0800] "GET /twiki/bin/search/Main/?scope=topic®ex=on&search=^g HTTP/1.1" 200 367564.242.88.10 - - [07/Mar/2004:17:17:27 -0800] "GET /twiki/bin/search/TWiki/?scope=topic®ex=on&search=^d HTTP/1.1" 200 5773lj1036.inktomisearch.com - - [07/Mar/2004:17:18:36 -0800] "GET /robots.txt HTTP/1.0" 200 68lj1090.inktomisearch.com - - [07/Mar/2004:17:18:41 -0800] "GET /twiki/bin/view/Main/LondonOffice HTTP/1.0" 200 386064.242.88.10 - - [07/Mar/2004:17:21:44 -0800] "GET /twiki/bin/attach/TWiki/TablePlugin HTTP/1.1" 401 1284664.242.88.10 - - [07/Mar/2004:17:22:49 -0800] "GET /twiki/bin/view/TWiki/ManagingWebs?rev=1.22 HTTP/1.1" 200 931064.242.88.10 - - [07/Mar/2004:17:23:54 -0800] "GET /twiki/bin/statistics/Main HTTP/1.1" 200 80864.242.88.10 - - [07/Mar/2004:17:26:30 -0800] "GET /twiki/bin/view/TWiki/WikiCulture HTTP/1.1" 200 593564.242.88.10 - - [07/Mar/2004:17:27:37 -0800] "GET /twiki/bin/edit/Main/WebSearch?t=1078669682 HTTP/1.1" 401 1284664.242.88.10 - - [07/Mar/2004:17:28:45 -0800] "GET /twiki/bin/oops/TWiki/ResetPassword?template=oopsmore¶m1=1.4¶m2=1.4 HTTP/1.1" 200 1128164.242.88.10 - - [07/Mar/2004:17:29:59 -0800] "GET /twiki/bin/view/TWiki/ManagingWebs?skin=print HTTP/1.1" 200 880664.242.88.10 - - [07/Mar/2004:17:31:39 -0800] "GET /twiki/bin/edit/Main/UvscanAndPostFix?topicparent=Main.WebHome HTTP/1.1" 401 1284664.242.88.10 - - [07/Mar/2004:17:35:35 -0800] "GET /twiki/bin/view/TWiki/KlausWriessnegger HTTP/1.1" 200 384864.242.88.10 - - [07/Mar/2004:17:39:39 -0800] "GET /twiki/bin/view/Main/SpamAssassin HTTP/1.1" 200 408164.242.88.10 - - [07/Mar/2004:17:42:15 -0800] "GET /twiki/bin/oops/TWiki/RichardDonkin?template=oopsmore¶m1=1.2¶m2=1.2 HTTP/1.1" 200 1128164.242.88.10 - - [07/Mar/2004:17:46:17 -0800] "GET /twiki/bin/rdiff/TWiki/AlWilliams?rev1=1.3&rev2=1.2 HTTP/1.1" 200 448564.242.88.10 - - [07/Mar/2004:17:47:43 -0800] "GET /twiki/bin/rdiff/TWiki/AlWilliams?rev1=1.2&rev2=1.1 HTTP/1.1" 200 523464.242.88.10 - - [07/Mar/2004:17:50:44 -0800] "GET /twiki/bin/view/TWiki/SvenDowideit HTTP/1.1" 200 3616
log
![Page 5: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/5.jpg)
log$ cat access.log | grep 401
![Page 6: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/6.jpg)
log$ cat access.log | grep 404
![Page 7: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/7.jpg)
log
![Page 8: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/8.jpg)
log
![Page 9: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/9.jpg)
tornando seu log útil…
![Page 10: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/10.jpg)
tornando seu log útil…
![Page 11: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/11.jpg)
200.164.237.13 - - [27/Aug/2015:12:37:38 -0300] "GET / HTTP/1.1" 200 763 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/601.1.39 (KHTML, like Gecko) Version/9.0 Safari/601.1.39"
![Page 12: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/12.jpg)
LOGSTASH
![Page 13: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/13.jpg)
LOGSTASH
ARMAZENAR
COLETAR DADOS DO LOG
MANIPULAR ENRIQUECER
![Page 14: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/14.jpg)
LOGSTASH
ARMAZENAR
COLETAR DADOS DO LOG
MANIPULAR ENRIQUECER
input
![Page 15: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/15.jpg)
LOGSTASH
ARMAZENAR
COLETAR DADOS DO LOG
MANIPULAR ENRIQUECER
input
Filter
![Page 16: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/16.jpg)
LOGSTASH
ARMAZENAR
COLETAR DADOS DO LOG
MANIPULAR ENRIQUECER
input
Filter
output
![Page 17: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/17.jpg)
LOGSTASHinput { stdin{}}
filter { mutate { add_field => {“_type” => “test” } }}
output { stdout { codec => rubydebug }}
![Page 18: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/18.jpg)
LOGSTASH
$ echo "qconrio 2015" | ./logstash/bin/logstash -f sample.conf
Logstash startup completed{ "message" => "qconrio 2015", "@version" => "1", "@timestamp" => "2015-08-24T03:41:13.956Z", "host" => "241191a9debd", "_type" => "meudoc"}Logstash shutdown completed
![Page 19: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/19.jpg)
LOGSTASH
input filter output
file
syslog
log4j
date
grok
geoip
S3
kafkaES
https://goo.gl/AbhrMihttps://goo.gl/2ofebshttps://goo.gl/oo7fMr
![Page 20: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/20.jpg)
tornando seu log útil…
![Page 21: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/21.jpg)
ELASTICSEARCH
![Page 22: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/22.jpg)
ELASTICSEARCH
DADOS E ANÁLISE EM TEMPO REAL ALTA DISPONIBILIDADE
MULTI-TENANCY FULL TEXT SEARCH
ORIENTADO A DOCUMENTOS SCHEMA FREE
RESTFUL API PERSISTÊNCIA POR OPERAÇÃO
![Page 23: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/23.jpg)
ELASTICSEARCH
BD Relacional Elasticsearchdatabase indextable type
row documentcolumn fieldschema mappingpartition shard
![Page 24: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/24.jpg)
ELASTICSEARCH
$ curl -X PUT http://localhost:9200/qcon/talk/1 -d ‘{ “name" : “Proatividade na analise de log com ELK”, “date" : “2015-08-27T16:45:00”, “city" : “Rio de Janeiro”}’
adicionar endpoint indice tipo id
documento
![Page 25: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/25.jpg)
ELASTICSEARCH
$ curl -X GET http://localhost:9200/qcon/talk/1
obter endpoint indice tipo id
![Page 26: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/26.jpg)
ELASTICSEARCHLOGSTASH
![Page 27: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/27.jpg)
ELASTICSEARCH
input { file{ path => “/var/log/apache2/access.log" }}
filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }}
output { elasticsearch { host => localhost }}
LOGSTASH
![Page 28: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/28.jpg)
ELASTICSEARCH
$ curl -X GET http://localhost:9200/logstash-*/_count
obter endpoint indice action
logstash-%{+YYYY.MM.dd}
![Page 29: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/29.jpg)
http://qcon.leo.sh
![Page 30: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/30.jpg)
ELASTICSEARCH{
"_index" : "logstash-2015.08.25", "_type" : "logs", "_id" : "AU9ik9_koi5WviutsXW2", "_score" : 1.0, “_source":{
"message":"186.194.65.168 - - [25/Aug/2015:01:58:21 +0000] \"GET /icons/ubuntu-logo.png HTTP/1.1\" 200 3688 \"http://qcon.leo.sh/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/601.1.39 (KHTML, like Gecko) Version/9.0 Safari/601.1.39\”",“@version":"1",“@timestamp":"2015-08-25T01:58:21.000Z",“host":"ip-172-31-31-206",“path":"/var/log/apache2/access.log",“clientip":"186.194.65.168",“ident":"-",“auth":"-","timestamp":"25/Aug/2015:01:58:21 +0000”,“verb":"GET",“request":"/icons/ubuntu-logo.png",“httpversion":"1.1",“response":"200",“bytes":"3688",“referrer":"\"http://qcon.leo.sh/\"","agent":"\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/601.1.39 (KHTML, like Gecko) Version/9.0 Safari/601.1.39\""}
}
![Page 31: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/31.jpg)
um pouco mais de dados…
![Page 32: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/32.jpg)
input { ...}
filter { ... geoip { source => "clientip" target => "geoip" database => "/opt/logstash/GeoLiteCity.dat" add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] }
mutate { convert => [ "[geoip][coordinates]", "float"] }}
output { ...}
ELASTICSEARCH LOGSTASH
![Page 33: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/33.jpg)
ELASTICSEARCH
{ "_index" : "logstash-2015.08.25", "_type" : "logs", "_id" : "AU9ik9_koi5WviutsXW2", "_score" : 1.0, “_source”:{
...“geoip":{“ip":"186.194.65.168",“country_code2":"BR",“country_code3":"BRA",“country_name":"Brazil",“continent_code":"SA",“latitude":-23.547699999999992,“longitude":-46.63579999999999,“location":[-46.63579999999999,-23.547699999999992],“coordinates”:[-46.63579999999999,-23.547699999999992]
} }
![Page 34: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/34.jpg)
tornando seu log útil…
![Page 35: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/35.jpg)
![Page 36: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/36.jpg)
KIBANA
Dashboard personalizados
Interface flexíveis
Exportar dados com facilidade
Análises sofisticadas
![Page 37: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/37.jpg)
KIBANA
![Page 38: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/38.jpg)
KIBANA
![Page 39: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/39.jpg)
KIBANA
![Page 40: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/40.jpg)
ambiente inicial
![Page 41: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/41.jpg)
ambiente atual
![Page 42: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/42.jpg)
ORGANIZE OS LOGS
VERIFIQUE O QUE É RELEVANTE
ENRIQUEÇA AS INFORMAÇÕES
FAÇA A ANÁLISE
centralização não é tudo!
![Page 43: QCon Rio 2015 - Proatividade na Análise de Logs com ELK](https://reader034.vdocuments.us/reader034/viewer/2022051710/58edb4d41a28abb8788b463d/html5/thumbnails/43.jpg)
obrigado.@leocomelli