Fluentd meets Beats Elasticsearch meetup #14 - Jan 7, 2015
Who are you? • Masahiro Nakagawa • github: @repeatedly • Treasure Data Inc. • Fluentd / td-agent developer • Fluentd Enterprise support • I love OSS :) • D Language, MessagePack, The organizer of several meetups, etc…
Beats • Agent for each purpose by Elastic • https://www.elastic.co/products/beats • official: topbeat, filebeat, packetbeat • 3rd party: dockerbeat, nginxbeat, etc… • Beats support several outputs: elasticsearch, logstash, stdout and etc. • logstash output uses lumberjack protocol so we can use it for communicating with Beats.
ﬂuent-plugin-beats • Input plugin for Elastic Beats • https://github.com/repeatedly/ﬂuent-plugin-beats • Use lumberjack protocol to handle events • Tested with topbeat, filebeat, packetbeat • Beats use same event format so it should work with 3rd party Beats.
Read nginx 100000 logs and count by ﬂowcounter_simple
Why filebeat is slow? 1. Lumberjack protocol doesn’t focus on throughput • lumberjack sends/receives ack on each record
data frame ack
ack Lumberjack protocol 2. Beats framework is slow? [Issue #587] • filebeat is slower than logstash-forwarder
Conclusion • Beats are useful for collecting various metrics • ﬂuent-plugin-beats can handle Beats event and route events to elasitcsearch properly • Thanks ﬂuent-plugin-elasticsearch plugin ;) • Note that filebeat is slow so it is not good on high volume environment • Use ﬂuentd or ﬂuent-agent-hydra instead