Who are you? • Masahiro Nakagawa • github: @repeatedly • Treasure Data Inc. • Fluentd / td-agent developer • https://jobs.lever.co/treasure-data • I love OSS :) • D Language, MessagePack, The organizer of several meetups, etc…
Logging for containers • Storage: • should be outside of containers / hosts • Transferring: • should be over network • Aggregation: • should be done per container / per service
Aggregation patterns 1) 1-level aggregation 2) 2-level aggregation Pros: Pros: • simple configuration • less connections Cons: • lower load for aggr • fixed aggr, address Cons: • many connections • more containers • high load for aggr
Aggregation patterns 1) 1-level aggregation 2) 2-level aggregation Apps sends logs There are some patterns over ﬂuent-logger - via udp / tcp - tail file or … - etc…
2-level aggr w/ Fluentd 1. Transfer logs over network 2. Container logs & tail generated log files
1) Network transferring • Apps sends logs: app • using ﬂuent-logger-* • over inter-container network over TCP • Pros: • no performance penalty • Cons: • docker logs is out of scope • a bit complex for logging
2) Tail container logs • Apps write logs to STDOUT app • docker writes it to log file • ﬂuentd container reads its file log file • Pros: • simple conf for apps & docker • logs include container logs • Cons: • in_tail performance penalty & a bit complex configuration http://www.ﬂuentd.org/guides/recipes/docker-logging
3) Logging driver • Apps write logs to STDOUT app • Docker sends its logs to ﬂuentd directly over TCP • Pros: • simple conf for apps & docker • Logs includes container logs • Cons: • Do you see any concerns?