Fluentd & Event logs Before: App server App server App server Application Application Application File File File ... File File File ... File File File ... File High latency must wait for a day Log server Hard to analyze complex text parsers Burst of traffic
Fluentd & Event logs After: App server App server App server Application Application Application Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd
# receive events via HTTP # save alerts to a file <source> <match alert.**> type http type file port 8888 path /var/log/fluent/alerts </source> </match> # read logs from a file # forward other logs to servers <source> # (load-balancing + fail-over) type tail <match **> path /var/log/httpd.log type forward format apache <server> tag apache.access host 192.168.0.11 </source> weight 20 </server> # save access logs to MongoDB <server> <match apache.access> host 192.168.0.12 type mongo weight 60 host 127.0.0.1 </server> </match> </match>
Fluentd vs Scribe • Deals with structured logs • Easy to install > “gem install fluentd” > apt-get and yum http://packages.treasure-data.com/ • Easy to customize • add/modify plugins without re-compiling > “gem search -rd fluent-plugin”
Fluentd vs Flume • Easy to setup > “sudo fluentd --setup && fluentd” • Very small footprint > small engine (3,000 lines) + plugins • JVM-free • Easy to configure
Plugins • Bundled plugins > file writes event logs to files hourly or daily > forward forwards event logs (+fail-over and load balancing) > exec passes event logs to/from external commands > tail reads event logs from a file (like `tail -f`)
Plugins • 3rd party plugins > scribe integrates Fluentd with Scribe > s3 uploads log files to Amazon S3 hourly or daily > mongo writes logs to MongoDB > hoop puts log files on Hadoop HDFS via Hoop ...
Plugin developer API • Unit test framework (like “MRUnit”) > Fluent::Test::InputTestDriver > Fluent::Test::OutputTestDriver > Fluent::Test::BufferedOutputTestDriver • Fluent::TailInput (base class of “tail” plugin) > text parser is customizable def parse_line(line)