logstash family introduction
DESCRIPTION
Logstash IntroductionTRANSCRIPT
Logstash Family Introduc4on
Owen
What is a log
• Oxford Dic4onary – a thick piece of wood that is cut from or has fallen from a tree
– (also logbook) an official record of events during a par4cular period of 4me, especially a journey on a ship
• 4me + data
In theory, life cycle of log
Record
Transmit
Analyze Store
Delete
In design, life cycle of log
Record
Transmit
Store
Delete
In fact, life cycle of log
Record
Delete
Problems
• Logging to a database or filesysytem • Logging has placed a load on the database and filesystem
• Mul4ple log formats • No easy way to search logs • No easy method to gather sta4s4cs
Find the logs of 16 computers 6 months ago ?
Why use Logstash?
• A lot choices! �• But we want a free & high-integrality &
easy to use solution • splunk (finding your faults, just like mom) • facebookarchive/scribe (2682 ★) • Graylog2(Server+WUI 1683 ★) • fluentd (2038 ★) • logstash (2689 ★)
logstash and other things
hRps://www.youtube.com/watch?v=RuUFnog29M4
Logstash
• Open Source, Apache Licence • WriRen in JRuby, Runs on JVM • Plugins easily wriRen in Ruby • Process mul4ple format ( input, output ) • Logstash Family! ( Elas4cSearch , Kibana)
LogStash Family architecture
Elas4cSearch
• A response to the claim : “Search is hard” • Powerful indexing & search tool • search & index data available Rescully as JSON over HTTP
Kibana
All-‐in-‐one!
How logstash works?
• logstash process events, not (only) loglines!
• “The logstash agent is a processing pipeline with 3 stages: – inputs -> filters -> outputs.” – separate threads
• “Inputs generate events, filters modify them, outputs ship them elsewhere.”
• -- [the life of an event in logstash] �
In my thinking, Event Life Cycle
Input
filter output
In fact, Event Life Cycle
event (Input -‐> output)
event -‐-‐-‐-‐-‐-‐-‐-‐-‐ input filter output
Logstash is a wooden tube
Input
Input
Input
filter
output
codec
filter filter
output
output
Logstash plugins Workflow
• inputs – How events get into LogStash.
• codecs – convert an incoming format into an internal representa4on
• filters – processing ac4ons on events : modify events or drop events
• outputs – How output events from LogStash
Logstash plugins
What is an event!?
• A @4mestamp (ISO 8601 4mestamp) • A messsage field ( data ) • A @version • host ( the host of sender) • type( syslog, irc, etc)
Exersice: Hello Word!
java -‐jar logstash-‐1.1.12-‐flatjar.jar agent -‐f hello.conf
java -‐jar logstash.jar agent -‐f hello.conf
Input
• tcp • udp • unix • file • syslog • redis • logstash-‐fowarder(former Lumberjack)
Codecs
• plain • json • rubydebug • mul4line
Outputs
• mongodb • elas4cSearch • email • file • jira
Exercise: Mul4ple input & output
logstash-‐forwader
• ♫ I'm a lumberjack and I'm ok! I sleep when idle, then I ship logs all day! I parse your logs, I eat the JVM agent for lunch! ♫
• WriRen in Go • lumberjack is reserved for protocol • Resource Usage Concerns • Need an SSL CA to verify the server
lumberjack
• Encryp4on & Authen4ca4on (TLS) • Compression ( reduce bandwidth) • Sequence & ack behavior like TCP • Low latency • Reliable Aplica4on-‐Level message transport
Forwarder Sample
Filters
• date • grok • drop • geoIP • mutate • mul4line
Exercise: Parse Data
filter config
powerful grok • Parse arbitrary text and structure it. • The syntax for a grok paRern is – %{SYNTAX:SEMANTIC}
• 55.3.244.1 GET /index.html 15824 – %{IP:client} – %{WORD:method} – %{URIPATHPARAM:request} – %{NUMBER:bytes}
• hRps://github.com/elas4csearch/logstash/blob/v1.4.2/paRerns/grok-‐paRerns
grok sample
drop
mutate
• Muta4ons on fields. – rename – remove – replace – join – split – upper – lower
mul4line
• Codecs & filter
Reference
• hRps://www.digitalocean.com/community/tutorials/how-‐to-‐use-‐logstash-‐and-‐kibana-‐to-‐centralize-‐and-‐visualize-‐logs-‐on-‐ubuntu-‐14-‐04
• hRp://www.vmdoh.com/blog/centralizing-‐logs-‐lumberjack-‐logstash-‐and-‐elas4csearch
• hRp://jpmens.net/2012/08/09/i-‐grok-‐how-‐to-‐mutate-‐a-‐file-‐with-‐logstash/
• hRp://gleenders.blogspot.tw/2014/02/logstash-‐glassfish.html