{"id":32466,"date":"2024-04-12T19:37:18","date_gmt":"2024-04-12T17:37:18","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/?p=32466"},"modified":"2024-09-10T17:25:56","modified_gmt":"2024-09-10T15:25:56","slug":"monitor-elasticsearch-cluster-with-zabbix","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/","title":{"rendered":"Monitor Elasticsearch Cluster with Zabbix"},"content":{"rendered":"\n

Setting up Zabbix monitoring over an Elasticsearch cluster is quiet easy as it does not require an agent install. As a matter a fact, the official template<\/a> uses the Elastic REST API. Zabbix server itself will trigger these requests.<\/p>\n\n\n\n

In this blog post, I will quick explain how to setup Elasticsearch cluster, then how easy the Zabbix setup is and list possible issues you might encounter.<\/p>\n\n\n\n

Elastic Cluster Setup<\/h2>\n\n\n\n

I will not go too much in detail as David<\/a> covered already many topics around ELK. Anyway, would you need any help to install, tune or monitor your ELK cluster fell free to contact us.<\/p>\n\n\n\n

My 3 virtual machines are provisioned with YaK<\/a> on OCI. Then, I install the rpm on all 3 nodes.<\/p>\n\n\n\n

After starting first node service, I am generating an enrollment token with this command:<\/p>\n\n\n

\n\/usr\/share\/elasticsearch\/bin\/elasticsearch-create-enrollment-token -node\n<\/pre><\/div>\n\n\n
This return a long string which I will need to pass on node 2 and 3 of the cluster (before starting anything):<\/p>\n\n\n
\n\/usr\/share\/elasticsearch\/bin\/elasticsearch-reconfigure-node --enrollment-token <...>\n<\/pre><\/div>\n\n\n
Output will look like that:<\/p>\n\n\n
\nThis node will be reconfigured to join an existing cluster, using the enrollment token that you provided.\nThis operation will overwrite the existing configuration. Specifically:\n  - Security auto configuration will be removed from elasticsearch.yml\n  - The [certs] config directory will be removed\n  - Security auto configuration related secure settings will be removed from the elasticsearch.keystore\nDo you want to continue with the reconfiguration process [y\/N]\n<\/pre><\/div>\n\n\n
After confirming with a y<\/code>, we are almost ready to start. First, we must update ES configuration file (ie. \/etc\/elasticsearch\/elasticsearch.yml<\/code>).<\/p>\n\n\n\n
    \n
  • Add IP of first node (only for first boot strapped) in cluster.initial_master_nodes: [\"10.0.0.x\"]<\/code><\/li>\n\n\n\n
  • Set listening IP of the inter-node trafic (to do on node 1 as well): transport.host: 0.0.0.0<\/code><\/li>\n\n\n\n
  • Set list of master eligible nodes: discovery.seed_hosts: [\"10.0.0.x:9300\"]<\/code><\/li>\n<\/ul>\n\n\n\n
    Now, we are ready to start node 2 and 3.<\/p>\n\n\n\n
    Let’s check the health of our cluster:<\/p>\n\n\n
    \ncurl -s https:\/\/localhost:9200\/_cluster\/health -k -u elastic:password | jq\n<\/pre><\/div>\n\n\n
    If you forgot elastic password, you can reset it with this command:<\/p>\n\n\n
    \n\/usr\/share\/elasticsearch\/bin\/elasticsearch-reset-password -u elastic\n<\/pre><\/div>\n\n\n
    Zabbix Configuration<\/h2>\n\n\n\n
    With latest Elasticsearch release, security has drastically increased as SSL communication became the standard. Nevertheless, default MACROS values of the template did not. Thus, we have to customize the followings:<\/p>\n\n\n\n
      \n
    • {$ELASTICSEARCH.USERNAME} to elastic<\/li>\n\n\n\n
    • {$ELASTICSEARCH.PASSWORD} to its password<\/li>\n\n\n\n
    • {$ELASTICSEARCH.SCHEME} to https<\/li>\n<\/ul>\n\n\n\n
      If SELinux is enabled on your Zabbix server, you will need to allow zabbix_server<\/code> process to send network request. Following command achieves this:<\/p>\n\n\n
      \nsetsebool zabbix_can_network 1\n<\/pre><\/div>\n\n\n
      Next, we can create a host in Zabbix UI like that:<\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      The Agent interface is required but will not be used to reach any agent as there are not agent based (passive or active) checks in the linked template. However, http checks uses HOST.CONN<\/mark> MACRO in the URLs. Ideally, the IP should be a virtual IP or a load balanced IP.<\/p>\n\n\n\n
      Don’t forget to set the MACROS:<\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      After few minutes, and once nodes discovery ran, you should see something like that:<\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      Problems<\/h2>\n\n\n\n
      What will happen if one node stops? On Problems <\/em>tab of Zabbix UI:<\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      After few seconds, I noticed that ES: Health is YELLOW<\/mark> gets resolved on its own. Why? Because shards are re-balanced across running servers.<\/p>\n\n\n\n
      I confirm this by graphing Number of unassigned shards<\/mark>:<\/p>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      We can also see the re-balancing with the network traffic monitoring:<\/p>\n\n\n\n
      \"\"
      Received bytes on the left. Sent on the right.<\/figcaption><\/figure>\n\n\n\n
      Around 15:24, I stopped node 3 and shards were redistributed from node 1 and 2.<\/p>\n\n\n\n
      When node 3 start, at 15:56, we can see node 1 and 2 (20 Mbps each) send back shards to node 3 (40 Mbps received).<\/p>\n\n\n\n
      Conclusion<\/h2>\n\n\n\n
      Whatever the monitoring tool you are using, it always help to understand what is happening behind the scene.<\/p>\n","protected":false},"excerpt":{"rendered":"
      Setting up Zabbix monitoring over an Elasticsearch cluster is quiet easy as it does not require an agent install. As a matter a fact, the official template uses the Elastic REST API. Zabbix server itself will trigger these requests. In this blog post, I will quick explain how to setup Elasticsearch cluster, then how easy […]<\/p>\n","protected":false},"author":40,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1320],"tags":[86,2430],"type_dbi":[],"class_list":["post-32466","post","type-post","status-publish","format-standard","hentry","category-devops","tag-elasticsearch","tag-zabbix"],"acf":[],"yoast_head":"\nMonitor Elasticsearch Cluster with Zabbix - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Monitor Elasticsearch Cluster with Zabbix\" \/>\n<meta property=\"og:description\" content=\"Setting up Zabbix monitoring over an Elasticsearch cluster is quiet easy as it does not require an agent install. As a matter a fact, the official template uses the Elastic REST API. Zabbix server itself will trigger these requests. In this blog post, I will quick explain how to setup Elasticsearch cluster, then how easy […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2024-04-12T17:37:18+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-09-10T15:25:56+00:00\" \/>\n<meta name=\"author\" content=\"Middleware Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Middleware Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/\"},\"author\":{\"name\":\"Middleware Team\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1\"},\"headline\":\"Monitor Elasticsearch Cluster with Zabbix\",\"datePublished\":\"2024-04-12T17:37:18+00:00\",\"dateModified\":\"2024-09-10T15:25:56+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/\"},\"wordCount\":505,\"commentCount\":0,\"keywords\":[\"Elasticsearch\",\"Zabbix\"],\"articleSection\":[\"DevOps\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/\",\"name\":\"Monitor Elasticsearch Cluster with Zabbix - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2024-04-12T17:37:18+00:00\",\"dateModified\":\"2024-09-10T15:25:56+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Monitor Elasticsearch Cluster with Zabbix\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1\",\"name\":\"Middleware Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"caption\":\"Middleware Team\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/middleware-team\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Monitor Elasticsearch Cluster with Zabbix - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/","og_locale":"en_US","og_type":"article","og_title":"Monitor Elasticsearch Cluster with Zabbix","og_description":"Setting up Zabbix monitoring over an Elasticsearch cluster is quiet easy as it does not require an agent install. As a matter a fact, the official template uses the Elastic REST API. Zabbix server itself will trigger these requests. In this blog post, I will quick explain how to setup Elasticsearch cluster, then how easy […]","og_url":"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/","og_site_name":"dbi Blog","article_published_time":"2024-04-12T17:37:18+00:00","article_modified_time":"2024-09-10T15:25:56+00:00","author":"Middleware Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Middleware Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/"},"author":{"name":"Middleware Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1"},"headline":"Monitor Elasticsearch Cluster with Zabbix","datePublished":"2024-04-12T17:37:18+00:00","dateModified":"2024-09-10T15:25:56+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/"},"wordCount":505,"commentCount":0,"keywords":["Elasticsearch","Zabbix"],"articleSection":["DevOps"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/","url":"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/","name":"Monitor Elasticsearch Cluster with Zabbix - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2024-04-12T17:37:18+00:00","dateModified":"2024-09-10T15:25:56+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/monitor-elasticsearch-cluster-with-zabbix\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Monitor Elasticsearch Cluster with Zabbix"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1","name":"Middleware Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","caption":"Middleware Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/middleware-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/32466","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/40"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=32466"}],"version-history":[{"count":10,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/32466\/revisions"}],"predecessor-version":[{"id":32546,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/32466\/revisions\/32546"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=32466"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=32466"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=32466"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=32466"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}