Metrics

Metrics are a numerical representation of data that can be used to determine a service or component’s overall behavior over time. Collecting metrics and creating alerts based on these metrics are key to do operations of any production deployment. All Kubernetes clusters at Gjensidige have the Kube Prometheus Stack installed by default, enabling developers to collect metrics in Prometheus 🔍, visualize them in Grafana 📺, and use them to trigger alerts with Alertmanager 🔔

Getting your metrics into Prometheus

Access to metrics must be restricted

For security reasons, end users should never be able to access application metrics. Metrics endpoints should only be available for internal systems 🔒

There are two recommended methods to instruct Prometheus to collect metrics from your Kubernetes workload:

Using a ServiceMonitor

A ServiceMonitor will instruct Prometheus to collect metrics from a port exposed with a Service. Let's assume you have the following Service resource:

service.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: "your-team-namespace"
  name: "test-app"
  labels:
    app: "test-app"
    environment: "test"
spec:
  selector:
    app: "test-app"
    environment: "test"
  type: ClusterIP
  ports:
    - port: 8081
      name: "metrics"
      protocol: TCP
      targetPort: "metrics"

Then you can instruct Prometheus to collect metrics from the service by specifying port name and endpoint:

servicemonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  namespace: "your-team-namespace"
  name: "test-app"
  labels:
    app: "test-app"
    environment: "test"
spec:
  endpoints:
    - interval: "15s"
      path: "/metrics" # Assuming the container running behind the service port exposes an endpoint '/metrics' with Prometheus formatted metrics
      port: "metrics"
  selector:
    matchLabels:
      app: "test-app"
      environment: "test"

Using a PodMonitor

A PodMonitor will instruct Prometheus to collect metrics directly from a container running inside a Pod. Let's assume you have the following Deployment resource:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: "your-team-namespace"
  name: "test-app"
  labels:
    app: "test-app"
    environment: "test"
spec:
  selector:
    matchLabels:
      app: "test-app"
      environment: "test"
  template:
    metadata:
      name: "test-app"
      labels:
        app: "test-app"
        environment: "test"
    spec:
      containers:
        - name: "test-app"
          image: "gjensidige.azurecr.io/test-app:12345"
          ports:
            - containerPort: 1337
              name: "metrics"
              protocol: TCP

Then you can instruct Prometheus to collect metrics directly from the container running by specifying port name and endpoint:

podmonitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  namespace: "your-team-namespace"
  name: "test-app"
  labels:
    app: "test-app"
    environment: "test"
spec:
  selector:
    matchLabels:
      app: "test-app"
      environment: "test"
  podMetricsEndpoints:
    - port: "metrics"
      path: "/metrics" # Assuming your container exposes an endpoint '/metrics' with Prometheus formatted metrics
      interval: "15s"

Useful Prometheus query examples

Prometheus has a nice Querying Basics guide which is recommended to start with. You might also find some inspiration from the following real-world query examples. These will work in Kubernetes clusters at Gjensidige:

# NGINX Ingress Controller - percentage of HTTP requests with status 5xx
100 * (sum(nginx_ingress_controller_requests{status=~"5.+"}) / sum(nginx_ingress_controller_requests))

# Current replicas for a HorizontalPodAutoscaler for a given workload
kube_horizontalpodautoscaler_status_current_replicas{job="<your-workload-name>", namespace="<your-namespace>"}