Kubernetes Metrics

Collect and ship Kubernetes container service metrics to Logstash and Elasticsearch

Metricbeat is a lightweight shipper that helps you monitor Kubernetes by collecting metrics from the containers running on the host system. Configure Metricbeat using the pre-defined examples below to collect and ship Kubernetes container service metrics and statistics to Logstash or Elasticsearch.

Install Integration

Please click on the Install Integration button to configure your stack for this source.

Copy the manifest file

Copy and use the Kubernetes Filebeat manifest below. If you aren't logged in, you may need to update the environment variables of your-logstash-host / your-logstash-port.

For use with version 7.x Filebeats.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: >-
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false
 
    metricbeat.autodiscover:
      providers:
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          unique: true
          templates:
            - config:
                - module: kubernetes
                  hosts: ["kube-state-metrics:8080"]
                  period: 10s
                  add_metadata: true
                  metricsets:
                    - state_node
                    - state_deployment
                    - state_daemonset
                    - state_replicaset
                    - state_pod
                    - state_container
                    - state_job
                    - state_cronjob
                    - state_resourcequota
                    - state_statefulset
                    - state_service
                - module: kubernetes
                  metricsets:
                    - apiserver
                  hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
                  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                  ssl.certificate_authorities:
                    - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                  period: 30s
 
    processors:
      - add_cloud_metadata:
 
    output.logstash:
      hosts: ['${LOGSTASH_HOST}:${BEATS_PORT}']
      loadbalance: true
      ssl.enabled: true
 
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  system.yml: >-
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        #- core
        #- diskio
        #- socket
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5      # include top 5 processes by CPU
        by_memory: 5   # include top 5 processes by memory
 
    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)'
  kubernetes.yml: >-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      host: ${NODE_NAME}
      hosts: ["https://${NODE_NAME}:10250"]
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.verification_mode: "none"
      # If there is a CA bundle that contains the issuer of the certificate used in the Kubelet API,
      # remove ssl.verification_mode entry and use the CA, for instance:
      #ssl.certificate_authorities:
        #- /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
    # Currently `proxy` metricset is not supported on Openshift, comment out section
 
    - module: kubernetes
      metricsets:
        - proxy
      period: 10s
      host: ${NODE_NAME}
      hosts: ["localhost:10249"]
 
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: metricbeat
  namespace: kube-system
  labels:
    k8s-app: metricbeat
spec:
  selector:
    matchLabels:
      k8s-app: metricbeat
  template:
    metadata:
      labels:
        k8s-app: metricbeat
    spec:
      serviceAccountName: metricbeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: metricbeat
          image: docker.elastic.co/beats/metricbeat:7.16.3
          args:
            - -c
            - /etc/metricbeat.yml
            - -e
            - -system.hostfs=/hostfs
          env:
            - name: LOGSTASH_HOST
              value: "@logstash.host"
            - name: BEATS_PORT
              value: "@logstash.sslPort"
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config
              mountPath: /etc/metricbeat.yml
              readOnly: true
              subPath: metricbeat.yml
            - name: data
              mountPath: /usr/share/metricbeat/data
            - name: modules
              mountPath: /usr/share/metricbeat/modules.d
              readOnly: true
            - name: proc
              mountPath: /hostfs/proc
              readOnly: true
            - name: cgroup
              mountPath: /hostfs/sys/fs/cgroup
              readOnly: true
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: cgroup
          hostPath:
            path: /sys/fs/cgroup
        - name: config
          configMap:
            defaultMode: 640
            name: metricbeat-daemonset-config
        - name: modules
          configMap:
            defaultMode: 640
            name: metricbeat-daemonset-modules
        - name: data
          hostPath:
            path: /var/lib/metricbeat-data
            type: DirectoryOrCreate
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metricbeat
subjects:
  - kind: ServiceAccount
    name: metricbeat
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: metricbeat
  apiGroup: rbac.authorization.k8s.io
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metricbeat
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: metricbeat
    namespace: kube-system
roleRef:
  kind: Role
  name: metricbeat
  apiGroup: rbac.authorization.k8s.io
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metricbeat-kubeadm-config
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: metricbeat
    namespace: kube-system
roleRef:
  kind: Role
  name: metricbeat-kubeadm-config
  apiGroup: rbac.authorization.k8s.io
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metricbeat
  labels:
    k8s-app: metricbeat
rules:
  - apiGroups:
      - ""
    resources:
      - nodes
      - namespaces
      - events
      - pods
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - replicasets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - apps
    resources:
      - statefulsets
      - deployments
      - replicasets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - batch
    resources:
      - jobs
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/stats
    verbs:
      - get
  - nonResourceURLs:
      - /metrics
    verbs:
      - get
 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: metricbeat
  namespace: kube-system
  labels:
    k8s-app: metricbeat
rules:
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - get
      - create
      - update
 
---

Deploy Pod

Now your deployment manifest is updated, you can deploy it using.

kubectl apply -f metricbeat-kubernetes.yaml

Confirm Deployment Completed

Confirm that your pod has successfully been deployed using:

kubectl --namespace=kube-system get ds/metricbeat  
kubectl --namespace=kube-system get pods

You should see a pod for each kubernetes node with a name similar to metricbeat-abcde listed. The pods should work though from Pending to Running within a couple of minutes as the containers are downloaded and started.

Check Logit.io for your logs

Data should now have been sent to your Stack.

View My Data

If you don't see take a look at How to diagnose no data in Stack below for how to diagnose common issues.

How to diagnose no data in Stack

If you don't see data appearing in your stack after following this integration, take a look at the troubleshooting guide for steps to diagnose and resolve the problem or contact our support team and we'll be happy to assist.

Compatibility

The Kubernetes module dashboards are tested with Kubernetes 1.13.0 and 1.14.0. Dashboards for controllermanager scheduler and proxy are not compatible with kibana versions below 7.2.0.

Kubernetes Module Dashboard

The Kubernetes module comes with predefined Kibana dashboards. To view your dashboards for any of your Logit.io stacks, launch Logs and choose Dashboards.

Prebuilt Kibana dashboards Controller manager Proxy Scheduler

Kubernetes MetricBeat Module Overview

The Metricbeat Module for Kubernetes is a purpose-built tool designed to facilitate the monitoring of Kubernetes clusters by aggregating a wide range of metrics. For those already familiar with Metricbeat's approach to data collection from various environments, its application to Kubernetes represents a seamless and powerful extension for in-depth Kubernetes monitoring.

Kubernetes, a leading container orchestration platform, is essential for managing containerized applications across multiple hosts, providing automation for deployment, scaling, and operations. The Metricbeat Module for Kubernetes interacts directly with the Kubernetes API to gather metrics related to nodes, pods, deployments, and more, offering comprehensive insights into cluster performance, resource utilization, and operational health.

This module has quickly become a go-to for DevOps engineers, system administrators, and Kubernetes operators, thanks to its straightforward configuration and the seamless integration with the ELK Stack (opens in a new tab). This integration enables real-time data monitoring, sophisticated analytics, and detailed visualization capabilities, offering a depth of understanding and operational visibility far beyond traditional monitoring solutions.

In contrast to other monitoring solutions that may rely on bespoke or less flexible formats, the Metricbeat Module for Kubernetes is tailored for ease of deployment, comprehensive analysis, and efficient troubleshooting within Kubernetes environments.

Our platform supports the Metricbeat Module for Kubernetes, providing centralized metrics management (opens in a new tab) designed for straightforward analysis and proactive monitoring of Kubernetes clusters. This eliminates the complexity of manually tracking and interpreting metrics across distributed environments, streamlining the monitoring process.

If you're seeking assistance with setting up the Metricbeat Module for Kubernetes or have any questions regarding the effective monitoring of your Kubernetes clusters, the Logit.io team is ready to assist. We invite you to visit our Help Centre or reach out to our support team via live chat and we'll be happy to assist.

Kafka Metrics NGINX Metrics