Prometheus

Prometheus

https://github.com/prometheus/prometheus

Download:

https://prometheus.io/download/#prometheus

How To Use Alertmanager And Blackbox Exporter To Monitor Your Web Server On Ubuntu 16.04

https://www.digitalocean.com/community/tutorials/how-to-use-alertmanager-and-blackbox-exporter-to-monitor-your-web-server-on-ubuntu-16-04

Enable Scraping in Kubernetes

Scraping is either enabled globally or explicitly.

For explicit scraping add the following annotation for Pods or services.

---
apiVersion: v1
kind: Service
metadata:
   name: my-service
   annotations:
     prometheus.io/scrape: "true"
     prometheus.io/path: "/metrics"            # optional
     prometheus.io/port: "9102"

Note that for Daemonsets you have to put the annotation in the template spec:

---
apiVersion: apps/v1beta2
kind: DaemonSet
spec:
  [...]
  template:
    metadata:
      [...]
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9102'

Query Syntax

From https://prometheus.io/docs/prometheus/latest/querying/examples/

http_requests_total
http_requests_total{job="apiserver", handler="/api/comments"}

# Range query for 5min
http_requests_total{job="apiserver", handler="/api/comments"}[5m]

# Pattern matching
http_requests_total{job=~".*server"}
http_requests_total{status!~"4.."}

# Aggregate and group by
sum(rate(http_requests_total[5m])) by (job)
sum by(a,b)(mymetric{field="value")

# Math
(instance_memory_limit_bytes - instance_memory_usage_bytes) / 1024 / 1024

# Top keys
topk(3, sum(rate(instance_cpu_time_ns[5m])) by (app, proc))

# Count and group
count(instance_cpu_time_ns) by (app)

Problems of Prometheus

• Issues with quantile calculation

Great Dashboards

• Grafana k8s Cost Report

Scaling Prometheus

Self-hosted

• Cortex, OpenSource
• M3DB, see also Quick Setup HowTo

SaaS

• GrafanaCloud
• Influx