elk devops
TRANSCRIPT
ELK, a real case study
Alessandro Mazzoli Sysadmin@Ideato [email protected]
Paolo Tonin Sysadmin@Ideato [email protected]
What is ELK stack?
Elasticsearch Logstash Kibana
http://en.wikipedia.org/wiki/Elk
Logging problems(especially if you are a small company)
Centralize, index, archive
Law constraints
Many log structures
Log As a Service?!?! Pretty expensive
Log analysis is hard for non technical people
cat access.log | cut -d' ' -f1 | sort | uniq
People don’t pay attention
to boring things.- Brain Rules by John Medina
Major logging systems
Pro
Why ELK, a case studyWeb Tier
ApacheNginx Proxy
MySQL DB
Proxy FE Database BE
NFS shared filesystem
We need to scale Web Tier
Why ELK, a case study
Nginx proxy LB
Apache web1 MySQL DB
Memcached, Logstash
Web TierProxy FE Database BE
Apache web2
NFS shared filesystem
Nginx proxy LB
Apache web1 MySQL DB
Memcached, Logstash
Web TierProxy FE Database BE
Apache web2
NFS shared filesystem
Why ELK, a case study
local log
local log
ELK architecture
Server 1
Log File
Logstash Shipper
Redis
Application
Logstash Indexer
Elasticsearch
Scale out any components!
Redis
Logstash Indexer
Redis
Redis
Redis
ElasticsearchElasticsearch
Server 2
Logstash Shipper
Server 1
Logstash Shipper
Logstash configurationinput {!
}
filter {!
}
output {!
}
Where log come from?
Logstash configurationinput {!
}
filter {!
}
output {!
}
How we threat them ?
Where log come from?
Logstash configurationinput {!
}
filter {!
}
output {!
}Where will be stored?
How we threat them ?
Where log come from?
Example configurationinput { file { path => "/var/log/messages" type => "syslog" } file { path => "/var/log/apache/access.log" type => “apache-access" }}
Example configurationfilter { if [type] =~ "access" { mutate { replace => { "type" => “apache-access” } } grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] } }}
Example configurationoutput { elasticsearch { host => localhost }}
“How can I collect and consult my application Magento log?”
Custom filter
Grok
It’s is currently the best way in logstash to parse crappy unstructured log data into something structured and queryable
- http://logstash.net/docs/1.4.2/filters/grok
https://github.com/ideatosrl/logstash-magento-filter
input { file { path => “/var/www/magento/var/log/*.log" type => "magento" codec => plain { charset => "ISO-8859-1" } }}
Logstash Shipper
https://github.com/ideatosrl/logstash-magento-filter
filter { if [type] == "magento" { grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{DATA:syslog_program}"} add_field => [ "received_at", "%{@timestamp}" ] } }}
Logstash Shipper
https://github.com/ideatosrl/logstash-magento-filter
output { redis { host => “REDIS_IP" data_type => "list" key => "logstash" }}
Logstash Shipper
https://github.com/ideatosrl/logstash-magento-filter
input { redis { host => "REDIS_IP" type => "redis-input" data_type => “list" key => "logstash" }}output { elasticsearch { host => “ES_PUBLIC_IP” protocol => "http" manage_template => false index => "logstash-%{+YYYY.MM.dd}" }}
Logstash Indexer
Before…2015-03-30T15:25:34.867Z SoapFault exception: [soap:Client] Server was unable to read request. ---> There is an error in XML document (2, 439). ---> Input string was not in a correct format. in /var/www/magento/vendor/connect20/MailUp/app/code/local/MailUp/MailUpSync/Model/Observer.php:158
{ "_index": "logstash-2015.03.30", "_type": "magento", "_id": "AUxrSbc5UWx9I25Cgios", "_score": null, "_source": { "message": "SoapFault exception: [soap:Client] Server was unable to read request. ---> There is an error in XML document (2, 435). ---> Input string was not in a correct format. in /var/www/magento/vendor/connect20/MailUp/app/code/local/MailUp/MailUpSync/Model/Observer.php:158", "@version": "1", "@timestamp": "2015-03-30T15:25:34.867Z", "type": "magento", "host": "web2", "path": "/var/www/magento/var/log/exception.log", },}
And display it!
“Should I use ELK?”
PRO• Easy to install
!
• Opensource !
• Not only web logs! !
• Many output filters; S3, Google Big Query, MongoDb etc…
ConclusionsCONS• Many moving parts, each part has
their problems and issues !
• Quickly evolution, prepare yourself to upgrade frequently !
• Out-of-the-box configurations are not suitable for medium/large deployment
Links
http://www.ideato.it/technical-articles/integrazione-logstash-magento
https://github.com/ideatosrl/logstash-magento-filter
https://www.youtube.com/watch?v=RuUFnog29M4
http://logstash.net
http://elastic.co
Questions?!?