got logs? get answers with elasticsearch elk - puppetconf 2014
DESCRIPTION
Got Logs? Get Answers with Elasticsearch ELK - Jordan Sissel, ElasticsearchTRANSCRIPT
![Page 1: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/1.jpg)
Got Logs?ELK stories and awesome.
@jordansissel #PuppetConf 2014
![Page 2: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/2.jpg)
Disclaimer
I apologize for any obnoxious animations in this presentation.JUST KIDDING. ANIMATIONS ARE AMAZING
![Page 3: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/3.jpg)
Hello friends!I work on Logstash
at Elasticsearch
![Page 4: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/4.jpg)
#PuppetApproved
APPROVED
![Page 5: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/5.jpg)
THE KING OF PAIN MOUNTAINRichard Pijnenburg !
Very Nice Human !
Puppet Specialist
👑
Twitter: @Richardp82 — Github and IRC: electrical
![Page 6: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/6.jpg)
Sorry about the previous slide.I got a little wild.
![Page 7: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/7.jpg)
![Page 8: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/8.jpg)
Always be testing!
![Page 9: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/9.jpg)
puppet testing tools?• rspec-puppet
• puppet-doc-lint
• puppet-lint
• beaker
![Page 10: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/10.jpg)
Elasticsearch 💚 Puppet
![Page 11: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/11.jpg)
Story time!
![Page 12: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/12.jpg)
Let’s talk about ELK in the Wild!
![Page 13: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/13.jpg)
“Oops it broke”
![Page 14: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/14.jpg)
![Page 15: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/15.jpg)
![Page 16: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/16.jpg)
![Page 17: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/17.jpg)
{ }
![Page 18: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/18.jpg)
Complex data at high volume is hard, but we can help.
![Page 19: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/19.jpg)
ELK @ Bloomberg
![Page 20: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/20.jpg)
need: 1.5 billion events per second
![Page 21: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/21.jpg)
need: logs from thousands of servers
![Page 22: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/22.jpg)
need: integration with in-house tools
![Page 23: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/23.jpg)
Rub some ELK on it!
Picture: Wikipedia - Richard Lydekker - Public Domain
![Page 24: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/24.jpg)
10+ departments using it
![Page 25: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/25.jpg)
ELK @
![Page 26: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/26.jpg)
50.56.197.244 - - [13/Sep/2012:02:34:37 -0400] "GET / HTTP/1.1" 200 41687 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent"!89.96.171.210 - - [13/Sep/2012:02:32:49 -0400] "GET /files/logstash/logstash-1.1.0-monolithic.jar HTTP/1.1" 200 40923996 "-" "Chef Client/0.10.10 (ruby-1.9.3-p194; ohai-0.6.4; amd64-freebsd8; +http://opscode.com)"!37.57.128.238 - - [13/Sep/2012:02:37:24 -0400] "GET / HTTP/1.1" 200 41687 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent"!199.21.99.109 - - [13/Sep/2012:02:38:12 -0400] "GET /blog/tags/packaging HTTP/1.1" 200 15152 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"!180.76.6.232 - - [13/Sep/2012:02:38:23 -0400] "GET /blog/tags/wrt54gl HTTP/1.1" 200 8867 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"!217.227.233.68 - - [13/Sep/2012:02:38:25 -0400] "GET /articles/ssh-security/ HTTP/1.1" 200 16543 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"!217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"!217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /style2.css HTTP/1.1" 200 4877 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"!217.227.233.68 - - [13/Sep/2012:02:38:26 -0400] "GET /images/jordan-80.png HTTP/1.1" 200 6146 "http://www.semicomplete.com/articles/ssh-security/" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"!217.227.233.68 - - [13/Sep/2012:02:38:31 -0400] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0"!184.73.137.50 - - [13/Sep/2012:02:38:28 -0400] "GET /files/logstash/logstash-1.1.1-monolithic.jar HTTP/1.1" 200 53813805 "-" "Chef Client/0.10.8 (ruby-1.8.7-p334; ohai-0.6.10; i686-linux; +http://opscode.com)"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/ HTTP/1.1" 200 4483 "http://news.ycombinator.com/item?id=4417660" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/bootstrap/css/bootstrap-responsive.min.css HTTP/1.1" 200 7680 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/css/style.css HTTP/1.1" 200 2715 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/css/jquery.ui.datepicker.css HTTP/1.1" 200 33035 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/excanvas.min.js HTTP/1.1" 200 19415 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/bootstrap/css/bootstrap.min.css HTTP/1.1" 200 71463 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.history.js HTTP/1.1" 200 6466 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/css/jquery-ui-1.8.16.custom.css HTTP/1.1" 200 50829 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.flot.min.js HTTP/1.1" 200 37554 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.flot.selection.min.js HTTP/1.1" 200 3532 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.smartresize.js HTTP/1.1" 200 1123 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/iso8601.min.js HTTP/1.1" 200 486 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/safebase64.js HTTP/1.1" 200 3264 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"!24.24.235.59 - - [13/Sep/2012:02:38:46 -0400] "GET /kibana/js/lib/jquery.min.js HTTP/1.1" 200 93868 "http://semicomplete.com/kibana/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1
![Page 27: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/27.jpg)
grep😱
![Page 28: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/28.jpg)
(?<a0>(?<a1>(?<a2>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b) +(?<a3>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?<a4>(?!<[0-9])(?<a5>(?:2[0123]|[01][0-9])):(?<a6>(?:[0-5][0-9]))(?::(?<a7>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9]))) (?<a8>(?:(?<a9>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b))|(?<a10>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])))) (?<a11>(?<a12>(?:[\w._/%-]+))(?:\[(?<a13>\b(?:[1-9][0-9]*)\b)\])?): (?<a14>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])):(?<a15>(?:[+-]?(?:[0-9]+))) \[(?<a16>(?<a17>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))/(?<a18>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b)/(?<a19>[0-9]+):(?<a20>(?!<[0-9])(?<a21>(?:2[0123]|[01][0-9])):(?<a22>(?:[0-5][0-9]))(?::(?<a23>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9])).(?<a24>(?:[+-]?(?:[0-9]+))))\] (?<a25>\S+) (?<a26>\S+)/(?<a27>\S+) (?<a28>(?:[+-]?(?:[0-9]+)))/(?<a29>(?:[+-]?(?:[0-9]+)))/(?<a30>(?:[+-]?(?:[0-9]+)))/(?<a31>(?:[+-]?(?:[0-9]+)))/(?<a32>\S+) (?<a33>(?:[+-]?(?:[0-9]+))) (?<a34>\S+) (?<a35>.*?) (?<a36>.*?) (?<a37>\S+) (?<a38>(?:[+-]?(?:[0-9]+)))/(?<a39>(?:[+-]?(?:[0-9]+)))/(?<a40>(?:[+-]?(?:[0-9]+)))/(?<a41>(?:[+-]?(?:[0-9]+)))/(?<a42>\S+) (?<a43>(?:[+-]?(?:[0-9]+)))/(?<a44>(?:[+-]?(?:[0-9]+))) \{(?<a45>(?<a46>.*?))\} \{(?<a47>(?<a48>.*?))\} "(?<a49>\b\w+\b) (?<a50>(?<a51>(?:/[A-Za-z0-9$.+!*'(){},~:;=#%_-]*)+)(?:(?<a52>\?[A-Za-z0-9$.+!*'(){},~#%&/=:;_-]*))?) HTTP/(?<a53>(?:(?<a54>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))")
![Page 29: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/29.jpg)
(?<a0>(?<a1>(?<a2>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b) +(?<a3>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])) (?<a4>(?!<[0-9])(?<a5>(?:2[0123]|[01][0-9])):(?<a6>(?:[0-5][0-9]))(?::(?<a7>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9]))) (?<a8>(?:(?<a9>\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b))|(?<a10>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])))) (?<a11>(?<a12>(?:[\w._/%-]+))(?:\[(?<a13>\b(?:[1-9][0-9]*)\b)\])?): (?<a14>(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])):(?<a15>(?:[+-]?(?:[0-9]+))) \[(?<a16>(?<a17>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))/(?<a18>\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b)/(?<a19>[0-9]+):(?<a20>(?!<[0-9])(?<a21>(?:2[0123]|[01][0-9])):(?<a22>(?:[0-5][0-9]))(?::(?<a23>(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)))(?![0-9])).(?<a24>(?:[+-]?(?:[0-9]+))))\] (?<a25>\S+) (?<a26>\S+)/(?<a27>\S+) (?<a28>(?:[+-]?(?:[0-9]+)))/(?<a29>(?:[+-]?(?:[0-9]+)))/(?<a30>(?:[+-]?(?:[0-9]+)))/(?<a31>(?:[+-]?(?:[0-9]+)))/(?<a32>\S+) (?<a33>(?:[+-]?(?:[0-9]+))) (?<a34>\S+) (?<a35>.*?) (?<a36>.*?) (?<a37>\S+) (?<a38>(?:[+-]?(?:[0-9]+)))/(?<a39>(?:[+-]?(?:[0-9]+)))/(?<a40>(?:[+-]?(?:[0-9]+)))/(?<a41>(?:[+-]?(?:[0-9]+)))/(?<a42>\S+) (?<a43>(?:[+-]?(?:[0-9]+)))/(?<a44>(?:[+-]?(?:[0-9]+))) \{(?<a45>(?<a46>.*?))\} \{(?<a47>(?<a48>.*?))\} "(?<a49>\b\w+\b) (?<a50>(?<a51>(?:/[A-Za-z0-9$.+!*'(){},~:;=#%_-]*)+)(?:(?<a52>\?[A-Za-z0-9$.+!*'(){},~#%&/=:;_-]*))?) HTTP/(?<a53>(?:(?<a54>(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))")
http://upload.wikimedia.org/wikipedia/commons/7/7f/Empty-frame.png
![Page 30: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/30.jpg)
![Page 31: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/31.jpg)
ELK @ CERN
![Page 32: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/32.jpg)
-Ykb2j2ojYU
![Page 33: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/33.jpg)
“CERN - Accelerating Science with Puppet - Tim Bell”
from PuppetConf 2012
![Page 34: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/34.jpg)
thousands of events per second
![Page 35: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/35.jpg)
- Gergo Horanyi @ CERN
“What we really liked about Kibana, that the application developers can create their own dashboards, and they can monitor their systems on their own, without any help from some other team”
![Page 36: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/36.jpg)
- Gergo Horanyi @ CERN
“Kibana is well done, usable by non-experts.”
![Page 37: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/37.jpg)
democratize your data
![Page 38: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/38.jpg)
![Page 39: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/39.jpg)
OpenStack elastic-recheck
“Use Elasticsearch to classify and track OpenStack gate failures"
![Page 40: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/40.jpg)
![Page 41: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/41.jpg)
![Page 42: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/42.jpg)
Online Gaming
![Page 43: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/43.jpg)
![Page 44: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/44.jpg)
“Feels like Logstash is being slow”
![Page 45: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/45.jpg)
http://upload.wikimedia.org/wikipedia/commons/5/56/India_Victor_Grigas_2011-13.jpg
![Page 46: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/46.jpg)
Yep, that’s a bug!
http://en.wikipedia.org/wiki/Scutelleridae
![Page 47: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/47.jpg)
This has a measured 6.3x perf improvement in grok filter performance.
![Page 48: Got Logs? Get Answers with Elasticsearch ELK - PuppetConf 2014](https://reader033.vdocuments.us/reader033/viewer/2022060110/5560cbd9d8b42a08088b4a53/html5/thumbnails/48.jpg)
Lots of success! Hurray!