Kafka Metrics

Collect and ship Kafka server metrics to Logstash and Elasticsearch

Metricbeat is a lightweight shipper that helps you monitor your Kafka servers by collecting metrics running on the Kafka server. Configure Metricbeat using the pre-defined examples below to collect and ship Apache Kafka service metrics and statistics to Logstash or Elasticsearch.

Install Integration

Please click on the Install Integration button to configure your stack for this source.

Install Metricbeat

To get started first follow the steps below:

Install metricbeat (opens in a new tab)
Root access
Verify the required port is open

Older versions can be found here: metricbeat 7 (opens in a new tab), metricbeat 6 (opens in a new tab), metricbeat 5 (opens in a new tab)

Copy Configuration File

The configuration file below is pre-configured to send data to your Logit.io Stack via Logstash.

Copy the configuration file below and overwrite the contents of metricbeat.yml.

###################### Logit.io Metricbeat Configuration #######################
# =========================== Modules configuration ============================
 
metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
 
  # Set to true to enable config reloading
  reload.enabled: false
 
  # Period on which files under path should be checked for changes
  #reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false
 
# ================================== Outputs ===================================
# ------------------------------ Logstash Output -------------------------------

⚠️

No input available! Your stack is missing the required input for this data source

Talk to support to add the input

Enable Module

There are several built-in metricbeat modules you can use. To enable the kafka module, run the following:

metricbeat modules list
metricbeat modules enable @module

Configure Module

Each module has its own configuration file where different metricsets can be enabled / disabled. Locate the configuration file for the kafka module.

/etc/metricbeat/modules.d/@module.yml

By default metricsets are disabled. To enable or disable a metric simply comment or uncomment the line out.

- module: kafka
  # metricsets:
    # - partition
    # - consumergroup
  period: 10s
  hosts: ["localhost:9092"]
  
  #client_id: metricbeat
  #retries: 3
  #backoff: 250ms

Start Metricbeat

Start or restart Metricbeat (opens in a new tab) to apply the configuration changes.

Check Logit.io for your logs

Data should now have been sent to your Stack.

View My Data

If you don't see take a look at How to diagnose no data in Stack below for how to diagnose common issues.

How to diagnose no data in Stack

If you don't see data appearing in your stack after following this integration, take a look at the troubleshooting guide for steps to diagnose and resolve the problem or contact our support team and we'll be happy to assist.

Compatibility

The Kafka modules are tested with Kafka 0.10.2.1, 1.1.0 and 2.1.1.

Kafka Module Dashboard

The Kafka module comes with predefined Kibana dashboards. To view your dashboards for any of your Logit.io stacks, launch Logs and choose Dashboards.

Prebuilt Kibana dashboard

Kafka Metricbeat Module Overview

The Metricbeat Module for Kafka is a specialized tool designed for the efficient monitoring of Kafka clusters by capturing a wide array of performance metrics. For those already familiar with Metricbeat's capabilities in aggregating data from different services, its extension to Kafka brings a familiar yet powerful monitoring solution to the table.

Kafka, a distributed streaming platform, plays a crucial role in data processing and real-time streaming analytics. The Metricbeat Module for Kafka leverages Kafka's JMX metrics to gather data on topics, partitions, brokers, and more, offering deep insights into throughput, latency, and system health.

This module is particularly valued by DevOps, data engineers, and system administrators tasked with overseeing Kafka clusters. Its straightforward setup and integration with the ELK Stack (opens in a new tab) enable real-time monitoring and analysis, significantly enhancing operational visibility and the ability to react swiftly to changes in system performance.

Compared to other monitoring methods that may rely on bespoke or rigid formats, the Metricbeat Module for Kafka is crafted to offer ease of analysis, streamlined troubleshooting, and comprehensive operational oversight for Kafka environments.

Our platform supports the Metricbeat Module for Kafka, offering centralized management of metrics that facilitates straightforward analysis and proactive monitoring. This support removes the complexity of manually tracking and interpreting metrics across distributed Kafka clusters.

Should you need help with configuring the Metricbeat Module for Kafka or have questions about monitoring your Kafka clusters effectively, the Logit.io team is ready to assist. We encourage you to visit our Help Centre or reach out to our support team via live chat and we'll be happy to assist.

HAProxy Metrics Kubernetes Metrics