Monitoring k8s
with Prometheus
Prometheus Tokyo Meetup #1

INTRODUCTION
ABOUT ME
@Ladicle
AYA IGARASHI
NTT Communications
Software Engineer

INTRODUCTION
AGENDA
Prometheus & Kubernetes Overview
01
features
Why use Prometheus?
02
k8s service discovery
Introduce good points
03
easy installation and powerful monitoring
Summarize
04
pros and cons

What is
Prometheus?

OVERVIEW
What is Prometheus?
Characteristics
● OSS monitoring & alerting tool
● Built at SoundCloud (inspired by Borgmon)
● Joined the
● Written in Go
● Pull model
Not Fit
● 100% accuracy, such as billing

OVERVIEW
Architecture

What is
Kubernetes?

OVERVIEW
What is Kubernetes(k8s)?
Characteristics
● OSS management tool of container
● Built at Google (inspired by Borg)
● Joined the CNCF
● Written in Go
● Latest version is 1.6 (Support RBAC)
Container Runtime
● Docker
● rkt
● frakit etc….

Why use
Prometheus?

OVERVIEW
Why use Prometheus?
Deciding factors
○ Containers are discovered via service discovery
○ Flexible query and dashboarding support
○ Smart alerting system such as grouping

OVERVIEW
Architecture
○ NodeExporter
■ VM(baremetal)
○ BlackboxExporter
■ HTTP/TCP/DNS
○ Kube-state-metrix
■ Kubernetes cluster

OVERVIEW
k8s Service Discovery
What can it do?
● Scrape node, pod, endpoint and service
○ Node: NodeExporter
○ Pod / Endpoint: CustomExporter
○ Service: BlackboxExporter
How to use it?
● Provided official prometheus chart
○ https://github.com/kubernetes/charts/tree/master/stable/prometheus

OVERVIEW
How to use it?
Scrape pods and endpoints
● Prometheus Server configuration
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '2379'
prometheus.io/metrics: /metrics

OVERVIEW
How to use it?
Scrape service with BlackboxExporter
● Probe endpoints over HTTP, HTTPS, DNS, TCP and ICMP.
○ https://github.com/prometheus/blackbox_exporter

OVERVIEW
Dashboard
How to install it?
● Provided official grafana chart
○ https://github.com/kubernetes/charts/tree/master/stable/grafana
Beautiful dashboards
● Published k8s dashboards
○ https://grafana.com/dashboards?search=kubernetes

OVERVIEW
Summarize
Cons.
● 日本で使っているところが少ない
● BlackboxExporterがchartsに入っていない
● K8s 1.6のRBACにchartsが対応していない
Pros.
● β版だが普通に動く(今後のupdateに注意)
● インストール周りが整備済み
● PromQLが強力
● アラートをまとめるのが楽

Thanks For Watching!
See you next time