fluentd - flexible, stable, scalable
TRANSCRIPT
![Page 1: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/1.jpg)
FluentdFlexible, Stable, Scalable
Suiting @Taipei.py
![Page 2: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/2.jpg)
ho am I
Suiting (@suitingtseng)
Gogolook Inc.
Data Team
![Page 3: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/3.jpg)
Before
![Page 4: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/4.jpg)
What is Fluentd?
• Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.
• Treasure Data: td-agent
![Page 5: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/5.jpg)
What is Fluentd?
• Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.
• Treasure Data: td-agent
![Page 6: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/6.jpg)
What is a log?
![Page 7: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/7.jpg)
Log definition
Time + Tag + Content
![Page 8: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/8.jpg)
After
![Page 9: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/9.jpg)
How?
• Lightweight: C + Ruby + MessagePack
• Pluggable architecture
• Built-in Reliability
![Page 10: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/10.jpg)
Input plugins
• forward
• tail
• AWS Simple Queue Service
• AWS CloudWatch
![Page 11: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/11.jpg)
input: tail$ cat /etc/td-‐agent/conf.d
<source>
type tail
path /var/log/nginx/access.log
pos_file /var/log/td-‐agent/httpd-‐access.log.pos
tag nginx.access
</source>
<match nginx.access>
blah blah
</match>
![Page 12: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/12.jpg)
input: forward$ cat /etc/td-‐agent/conf.d
<source>
type forward
port 24224
</source>
<match flask.index>
blah blah
</match>
![Page 13: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/13.jpg)
input: forward$ cat ~/example.py
from fluent import sender
from fluent import event
sender.setup('flask', host='localhost', port=24224)
event.Event("index", {
"user": "foo",
"token": "bar",
"action": "POST"
})
![Page 14: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/14.jpg)
Output plugins
• forward
• copy
• Elasticsearch / MongoDB
• statsd / influxDB / graphite
• S3 / GCS / BigQuery
![Page 15: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/15.jpg)
output: elasticsearch$ cat /etc/td-‐agent/conf.d
<source>
foo bar
tag nginx.access
</source>
<match nginx.access>
type elasticsearch
hosts es-‐host1,es-‐host2
index_name nginx
type_name access
flush_interval 60s
</match>
![Page 16: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/16.jpg)
output: splunk$ cat /etc/td-‐agent/conf.d
<source>
foo bar
tag nginx.access
</source>
<match nginx.access>
type splunk
hosts splunk-‐host1
</match>
![Page 17: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/17.jpg)
Filter plugins
• grok
• grep
• record-modifier / record-reformer
• geoip
![Page 18: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/18.jpg)
Buffer types
• Memory
• File
![Page 19: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/19.jpg)
Buffer example$ cat /etc/td-‐agent/conf.d
<source>
foo bar
tag nginx.access
</source>
<match nginx.access>
type splunk
hosts splunk-‐host1
buffer_chunk_limit 10m
buffer_queue_limit 1000
flush_interval 5m
</match>
![Page 20: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/20.jpg)
![Page 21: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/21.jpg)
Scalability
• Scale up: multi-process plugin
• Scale out: out-forward plugin
![Page 22: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/22.jpg)
App + Fluentd
Fluentd
Elastic search
Elastic search
Elastic search
Elastic search
App + Fluentd
App + Fluentd
![Page 23: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/23.jpg)
Fluentd
Elastic search
Elastic search
Elastic search
Elastic search
Fluentd
App + Fluentd
App + Fluentd
App + Fluentd
![Page 24: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/24.jpg)
Fluentd
Elastic search
Elastic search
Elastic search
Elastic search
Fluentd
FluentdLoad
balance
App + Fluentd
App + Fluentd
App + Fluentd
Auto scaling group
![Page 25: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/25.jpg)
Stability
• Auto retry
• Persistent file buffer
• At-most-once delivery
![Page 26: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/26.jpg)
Message Delivery
• At-most-once: data may be lost
• At-least-once: data may be duplicated
• Exactly-once: perfect
![Page 27: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/27.jpg)
Idempotent
• HTTP PUT
• Maintain a unique id in application level or
• Concatenate (instance-id, time, ….) as id
![Page 28: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/28.jpg)
Gogolook use cases
• MongoDB, nginx log
• API, worker log
• Monitor
• Benchmark
![Page 29: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/29.jpg)
Active users by day
![Page 30: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/30.jpg)
System monitor
![Page 31: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/31.jpg)
Queue monitor
![Page 32: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/32.jpg)
Benchmark?
FluentdApp + Fluentd DB
![Page 33: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/33.jpg)
Benchmark?
FluentdApp + Fluentd DB
Local files
![Page 34: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/34.jpg)
Benchmark?
FluentdApp + Fluentd DB
Local files
![Page 35: Fluentd - Flexible, Stable, Scalable](https://reader036.vdocuments.us/reader036/viewer/2022081605/58f9a97a760da3da068b6f08/html5/thumbnails/35.jpg)
Q & A