1

Logstash Pattern - DTM

2

Goal

1) Ensure that developers can quickly get comprehensive log and performance data without operations intervention.

2) Avoid a complete rewrite of code.

3) Avoid brittle regexp parsing that may change often.

3

Solution – DTM

● - DTM stands for nothing● - Just a short string that rarely appears in the languages that I support● - Good intermediate step

4

Fixes...

Log in JSON

- parsing is cheap

- format change is fine

5

Example Usage

Calls to some 3rd party API are failing intermittently and/or taking too long

How do I figure out what is going on when there are several hundred requests a second and there are 40 app servers?

I don't have the time to re-write all of the logging code to use JSON

6

Answer...

Use DTM_EVENTS and DTM_STATS

7

Sample Code

8

Outputs the following log lines

2013-11-14T23:36:23.491037+00:00 app[web.1]: {"DTM_EVENT":"Failure","fail":"BROKEN","user":"chris","group":"admins"}

2013-11-14T23:36:43.080817+00:00 app[web.1]: {"DTM_STATS":"outside_api.fetch","value":0.089154052734,"stat_type":"timer"}

9

DTM_EVENTS

-JSON Blob

-Needs a unique DTM_EVENT field

{"DTM_EVENT":"Failure",

"fail":"BROKEN",

"user":"chris",

"group":"admins"}

10

Example Config (pre 1.3)

11

12

DTM_STATS

- JSON blobs

{"DTM_STATS":"outside_api.fetch",

"value":0.0001437664031982422,

"stat_type":"timer"}

Required fields

1) DTM_STATS – stat name

2) value number

3) stat_type

timer, counter, gauge

13

DTM_STATS types

Timer -

Timed events

Statsd provides the following data count_ps count lower mean_90 mean median std sum_90 sum upper_90 upper

Counter -

Rate events eg: number of HTTP 200's a second

Gauge -

Number during a given time period eg: 7 200's in the past minute

14

Example Config (pre 1.3)

15

Can be found in graphite

stats.timers.dtm_stats.application.heroku.staging.outside_api.fetch.mean_90

stats.guage.dtm_stats.application.heroku.staging.STAT

stats.counter.dtm_stats.application.heroku.staging.STAT

16

Examples of Graphite GraphsCan be saved under UserGraphs

17

More Information

Graphite on EC2

http://shokunin.co/blog/2013/03/16/graphite_on_ec2.html

Elasticsearch for Logstash

http://shokunin.co/blog/2013/06/24/elasticsearch.html

Map visualizations

http://shokunin.co/blog/2013/05/10/maptail_plus_logstash.html

18

Thanks!

Chris Mague

Shokunin.co

chris.mague@shokunin.co

IRC - #bosszaru

AIM - maguech