elk devops

ELK, a real case study

Alessandro Mazzoli Sysadmin@Ideato am@ideato.it

Paolo Tonin Sysadmin@Ideato pt@ideato.it

What is ELK stack?

Elasticsearch Logstash Kibana

http://en.wikipedia.org/wiki/Elk

Logging problems(especially if you are a small company)

Centralize, index, archive

Law constraints

Many log structures

Log As a Service?!?! Pretty expensive

Log analysis is hard for non technical people

cat access.log | cut -d' ' -f1 | sort | uniq

People don’t pay attention

to boring things.- Brain Rules by John Medina

Major logging systems

Pro

Why ELK, a case studyWeb Tier

ApacheNginx Proxy

MySQL DB

Proxy FE Database BE

NFS shared filesystem

We need to scale Web Tier

Why ELK, a case study

Nginx proxy LB

Apache web1 MySQL DB

Memcached, Logstash

Web TierProxy FE Database BE

Apache web2

NFS shared filesystem

Nginx proxy LB

Apache web1 MySQL DB

Memcached, Logstash

Web TierProxy FE Database BE

Apache web2

NFS shared filesystem

Why ELK, a case study

local log

local log

ELK architecture

Server 1

Log File

Logstash Shipper

Redis

Application

Logstash Indexer

Elasticsearch

Scale out any components!

Redis

Logstash Indexer

Redis

Redis

Redis

ElasticsearchElasticsearch

Server 2

Logstash Shipper

Server 1

Logstash Shipper

Logstash configurationinput {!

}

filter {!

}

output {!

}

Where log come from?

Logstash configurationinput {!

}

filter {!

}

output {!

}

How we threat them ?

Where log come from?

Logstash configurationinput {!

}

filter {!

}

output {!

}Where will be stored?

How we threat them ?

Where log come from?

Example configurationinput { file { path => "/var/log/messages" type => "syslog" } file { path => "/var/log/apache/access.log" type => “apache-access" }}

Example configurationfilter { if [type] =~ "access" { mutate { replace => { "type" => “apache-access” } } grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] } }}

Example configurationoutput { elasticsearch { host => localhost }}

“How can I collect and consult my application Magento log?”

Custom filter

Grok

It’s is currently the best way in logstash to parse crappy unstructured log data into something structured and queryable

- http://logstash.net/docs/1.4.2/filters/grok

https://github.com/ideatosrl/logstash-magento-filter

input { file { path => “/var/www/magento/var/log/*.log" type => "magento" codec => plain { charset => "ISO-8859-1" } }}

Logstash Shipper

https://github.com/ideatosrl/logstash-magento-filter

filter { if [type] == "magento" { grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{DATA:syslog_program}"} add_field => [ "received_at", "%{@timestamp}" ] } }}

Logstash Shipper

https://github.com/ideatosrl/logstash-magento-filter

output { redis { host => “REDIS_IP" data_type => "list" key => "logstash" }}

Logstash Shipper

https://github.com/ideatosrl/logstash-magento-filter

input { redis { host => "REDIS_IP" type => "redis-input" data_type => “list" key => "logstash" }}output { elasticsearch { host => “ES_PUBLIC_IP” protocol => "http" manage_template => false index => "logstash-%{+YYYY.MM.dd}" }}

Logstash Indexer

Before…2015-03-30T15:25:34.867Z SoapFault exception: [soap:Client] Server was unable to read request. ---> There is an error in XML document (2, 439). ---> Input string was not in a correct format. in /var/www/magento/vendor/connect20/MailUp/app/code/local/MailUp/MailUpSync/Model/Observer.php:158

{ "_index": "logstash-2015.03.30", "_type": "magento", "_id": "AUxrSbc5UWx9I25Cgios", "_score": null, "_source": { "message": "SoapFault exception: [soap:Client] Server was unable to read request. ---> There is an error in XML document (2, 435). ---> Input string was not in a correct format. in /var/www/magento/vendor/connect20/MailUp/app/code/local/MailUp/MailUpSync/Model/Observer.php:158", "@version": "1", "@timestamp": "2015-03-30T15:25:34.867Z", "type": "magento", "host": "web2", "path": "/var/www/magento/var/log/exception.log", },}

And display it!

“Should I use ELK?”

PRO• Easy to install

!

• Opensource !

• Not only web logs! !

• Many output filters; S3, Google Big Query, MongoDb etc…

ConclusionsCONS• Many moving parts, each part has

their problems and issues !

• Quickly evolution, prepare yourself to upgrade frequently !

• Out-of-the-box configurations are not suitable for medium/large deployment

Links

http://www.ideato.it/technical-articles/integrazione-logstash-magento

https://github.com/ideatosrl/logstash-magento-filter

https://www.youtube.com/watch?v=RuUFnog29M4

http://logstash.net

http://elastic.co

Questions?!?