fluentd v1 and roadmap
DESCRIPTION
Fluentd Meetup at FreakOut.TRANSCRIPT
![Page 1: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/1.jpg)
May 13, 2014
www.treasuredata.com/
Fluentd v1 and Roadmap
Masahiro Nakagawa Treasure Data, Inc
1
![Page 2: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/2.jpg)
Who are you?
• Masahiro Nakagawa
• @repeatedly
• Treasure Data, Inc.
• Senior Software Engineer
• Fluentd, td-agent, etc...
• Dlang, MessagePack, ...
2
![Page 3: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/3.jpg)
Structured logging !
Reliable forwarding !
Pluggable architecture
http://fluentd.org/
![Page 4: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/4.jpg)
M x N → M + N
4
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databasesbuffer / buffer / routing
![Page 5: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/5.jpg)
v10 (gem v0.10.x)
• Mainly for log forwarding
• with good performance
• working in production (at Nintendo, etc.)
• Various plugins are released
• There are 250+ plugins!
• Mainly for CRuby
5
![Page 6: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/6.jpg)
v11 is gone…
• Started two years ago
• Required dramatic API changes…
• What does ‘v11’ mean?
• Where is ‘v10’?
• See the following article
6
http://repeatedly.github.io/ja/2014/03/about-fluentd-v11/
![Page 7: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/7.jpg)
v1 is coming!
• Merge useful v11 features
• No breaking API compatibility
• Will go over features in later slides
• Clear versioning and stability
• https://github.com/fluent/fluentd/issues/251
7
![Page 8: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/8.jpg)
Must features
8
![Page 9: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/9.jpg)
New configuration
• Can use Hash, Array and others
• No need for “,” or similar tricks
• You can write Ruby code directly
• Can use via “--use-v1-config” option
• available since v0.10.46
• Worker pragma (nice to have)
9
![Page 10: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/10.jpg)
• Can write complex values without DSL!
• Can use Ruby code for configuration
New parameter types
10
<source> type my_tail keys ["k1", "k2", "k3"] </source> !<match **> typo my_filter add_keys {"k1" : "v1"} </match>
<match **> type my_filter env "#{ENV['KEY']}" </match>
Hash, Array, etc: Embedded Ruby code:
• Socket.gethostname
• `command`
• etc...
![Page 11: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/11.jpg)
Filter / Label support
• No more tag-related tricks!
• add_tag_xxx, remove_tag_xxx, etc...
!
• Redirect events to another group
• Much easier to group and share plugins
11
![Page 12: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/12.jpg)
Filter
• <match> can have nested <match>
• Configuration format is not fixed!
12
<match access.**> type flowcounter add_tag_prefix counted </match> !<match counted.**> type growthforecast </match>
<match access.** copy> type flowcounter ! <match **> type growthforecast </match> </match>
v10: v1:
![Page 13: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/13.jpg)
Label
• <label> can contain multiple <match>
• out_redirect can forward events to <label>
13
<match access.**> type rewrite_tag_filter ... <match bang.**> type redirect to_label blackhole </match> ... </match>
<label blackhole> <match **> type null </match> </label>
bang’s records go away!
![Page 14: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/14.jpg)
Improved plugin
• Error stream
• with @ERROR label
• No more global API, Engine.emit, $log, etc...
• Log level per plugin
• available since v0.10.43
• Actor (nice to have)
14
![Page 15: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/15.jpg)
ERROR!
Error stream
• Can handle an error at each record level
15
{"event":1, ...}
{"event":2, ...}
{"event":3, ...}
chunk1
{"event":4, ...}
{"event":5, ...}
{"event":6, ...}
chunk2
…
Input
OK
ERROR!
OK
OK
OK
Output
<label @ERROR> <match **> type file ... </match> </label>
Error stream
Built-in @ERROR is used when error occurred in “emit”
![Page 16: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/16.jpg)
Nice to have features
16
![Page 17: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/17.jpg)
ServerEngine based
• Robust signal handling
• Put a signal into Queue first
• Built-in supervisor
• Multiprocess support
• No need for in_multiprocess plugin
17
![Page 18: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/18.jpg)
Multi Process
18
Worker
Supervisor
Worker Worker
Separate stream pipelines in one instance!
<worker> input tail output mongo </worker>
<worker> input forward output webhdfs </worker>
<worker> input foo output bar </worker>
![Page 19: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/19.jpg)
• SocketManager shares resources with workers
Zero downtime restart
19
SupervisorTCP
1. Listen to TCP socket
![Page 20: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/20.jpg)
• SocketManager shares resources with workers
Zero downtime restart
20
Worker
Supervisor
heartbeat
TCP
TCP
1. Listen to TCP socket
2. Pass its socket to worker
![Page 21: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/21.jpg)
• SocketManager shares resources with workers
Zero downtime restart
21
Worker
Supervisor
Worker
TCP
TCP
1. Listen to TCP socket
2. Pass its socket to worker
3. Do same actionat worker restartingwith keeping TCP socket
heartbeat
![Page 22: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/22.jpg)
Actor• Easy to write popular routines
• Hide the implementation details
22
class TimerWatcher < Coolio::TimerWatcher ... end !def start @loop = Coolio::Loop.new @timer = ... @loop.attach(@timer) @thread = ... end
actor.every(@interval) { event_router.emit(...) }
v10: v1:
![Page 23: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/23.jpg)
Others
23
![Page 24: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/24.jpg)
JRuby and Windows
• Windows support
• Need testers!
• https://github.com/fluent/fluentd/tree/windows
• JRuby support
• https://github.com/fluent/fluentd/issues/317
24
![Page 25: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/25.jpg)
td-agent2
25
• Use Ruby 2.1.2
• Update core libraries
• msgpack, cool.io and etc
• Use v1 config by default
• http://docs.fluentd.org/articles/config-file#v1-format
![Page 26: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/26.jpg)
26New website…?
![Page 27: Fluentd v1 and Roadmap](https://reader033.vdocuments.us/reader033/viewer/2022052505/554be914b4c9056b348b4db7/html5/thumbnails/27.jpg)
Conclusion
27
• Fluend advances to the next stage!
• v1 will be released
• No breaking API compatibility
• td-agent2 will be released in this month
• New website with good contents?
• Need patches or feedback!