Elasticsearch Metrics

Ship your Elasticsearch Metrics via Telegraf to your Logit.io Stack

Configure Telegraf to ship Elasticsearch metrics to your Logit.io stacks via Logstash.

Install Integration

Please click on the Install Integration button to configure your stack for this source.

Install Telegraf

This integration allows you to configure a Telegraf agent to send your metrics, in multiple formats, to Logit.io.

Choose the installation method for your operating system:

When you paste the command below into Powershell it will download the Telegraf zip file. Once that is complete, press Enter again and the zip file will be extracted into C:\Program Files\InfluxData\telegraf\telegraf-1.31.2.

wget https://dl.influxdata.com/telegraf/releases/telegraf-1.31.2_windows_amd64.zip -UseBasicParsing -OutFile telegraf-1.31.2_windows_amd64.zip 
Expand-Archive .\telegraf-1.31.2_windows_amd64.zip -DestinationPath 'C:\Program Files\InfluxData\telegraf'

Configure the Telegraf input plugin

The configuration file below is pre-configured to scrape the system metrics from your hosts, add the following code to the configuration file /etc/telegraf/telegraf.conf from the previous step.

[[inputs.elasticsearch]]
  ## specify a list of one or more Elasticsearch servers
  ## you can add username and password to your url to use basic authentication:
  ## servers = ["http://user:pass@localhost:9200"]
  servers = ["http://localhost:9200"]
 
  ## Timeout for HTTP requests to the elastic search server(s)
  http_timeout = "5s"
 
  ## When local is true (the default), the node will read only its own stats.
  ## Set local to false when you want to read the node stats from all nodes
  ## of the cluster.
  local = true
 
  ## Set cluster_health to true when you want to obtain cluster health stats
  cluster_health = false
 
  ## Adjust cluster_health_level when you want to obtain detailed health stats
  ## The options are
  ##  - indices (default)
  ##  - cluster
  # cluster_health_level = "indices"
 
  ## Set cluster_stats to true when you want to obtain cluster stats.
  cluster_stats = false
 
  ## Only gather cluster_stats from the master node. To work this require local = true
  cluster_stats_only_from_master = true
 
  ## Indices to collect; can be one or more indices names or _all
  ## Use of wildcards is allowed. Use a wildcard at the end to retrieve index names that end with a changing value, like a date.
  indices_include = ["_all"]
 
  ## One of "shards", "cluster", "indices"
  ## Currently only "shards" is implemented
  indices_level = "shards"
 
  ## node_stats is a list of sub-stats that you want to have gathered. Valid options
  ## are "indices", "os", "process", "jvm", "thread_pool", "fs", "transport", "http",
  ## "breaker". Per default, all stats are gathered.
  # node_stats = ["jvm", "http"]
 
  ## HTTP Basic Authentication username and password.
  # username = ""
  # password = ""
 
  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false
 
  ## Sets the number of most recent indices to return for indices that are configured with a date-stamped suffix.
  ## Each 'indices_include' entry ending with a wildcard (*) or glob matching pattern will group together all indices that match it, and ## sort them by the date or number after the wildcard. Metrics then are gathered for only the 'num_most_recent_indices' amount of most ## recent indices.
  # num_most_recent_indices = 0

Read more about how to configure data scraping and configuration options for Elasticsearch (opens in a new tab)

Configure The Output plugin

Once you have generated the configuration file, you need to set up the output plug-in to allow Telegraf to transmit your data to Logit.io in Prometheus format. This can be accomplished by incorporating the following code into your configuration file:

[[outputs.http]]
  url = "https://@metricsUsername:@metricsPassword@@metrics_id-vm.logit.io:@vmAgentPort/api/v1/write"
  data_format = "prometheusremotewrite"
 
  [outputs.http.headers]
    Content-Type = "application/x-protobuf"
    Content-Encoding = "snappy"

Start Telegraf

From the location where Telegraf was installed (C:\Program Files\InfluxData\telegraf\telegraf-1.31.2) run the program providing the chosen configuration file as a parameter:

.\telegraf.exe --config telegraf-demo.conf

Once Telegraf is running you should see output similar to the following, which confirms the inputs, output and basic configuration the application has been started with: Powershell Telegraf information

View your metrics

Data should now have been sent to your Stack.

View My Data

If you don't see take a look at How to diagnose no data in Stack below for how to diagnose common issues.

How to diagnose no data in Stack

If you don't see data appearing in your Stack after following the steps, visit the Help Centre guide for steps to diagnose no data appearing in your Stack or Chat to support now.

Telegraf Elasticsearch metrics Overview

To effectively monitor and analyze Elasticsearch metrics in a distributed environment, it's crucial to leverage a dependable and efficient metrics management solution. Telegraf, an open-source metrics collection agent, is an excellent choice for this task, capable of collecting Elasticsearch metrics from a variety of sources, including operational Elasticsearch clusters, databases, and other relevant applications.

Telegraf offers a diverse range of input plugins, enabling users to gather metrics from a multitude of sources, such as CPU usage, memory consumption, network activity, and more. To store and interrogate these collected metrics, organizations can utilize Prometheus, an open-source monitoring and alerting system known for its flexible querying language and robust graphical data visualization capabilities.

To ship Elasticsearch metrics from Telegraf to Prometheus, organizations need to configure Telegraf to output metrics in the Prometheus format, and then set up Prometheus to scrape these metrics from the Telegraf server. This process involves setting up Telegraf to collect Elasticsearch metrics and output them in the Prometheus format, configuring Prometheus to fetch these metrics from the Telegraf server, and then visually analyzing the data using Prometheus's flexible querying and graphical visualization tools.

Once the metrics have been successfully ingested into Prometheus, further analysis and visualization can be performed using Grafana. Grafana is an open-source platform renowned for its monitoring and observability features, and is fully compatible with Prometheus. It allows users to create dynamic, interactive dashboards, offering a deeper understanding of the metrics data and a more comprehensive grasp of performance trends and potential issues.

If you need any further assistance with shipping your log data to Logit.io we're here to help you get started. Feel free to get in contact with our support team by sending us a message via live chat & we'll be happy to assist.