{"id":24533,"date":"2023-04-18T10:52:51","date_gmt":"2023-04-18T08:52:51","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/?p=24533"},"modified":"2024-09-11T10:18:47","modified_gmt":"2024-09-11T08:18:47","slug":"kafka-monitoring-in-prometheus","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/","title":{"rendered":"Kafka Monitoring in Prometheus"},"content":{"rendered":"\n<p>Enabling monitoring on Kafka with Prometheus\/Grafana is not more difficult than others exporters I configured in my previous <em>monitoring in Prometheus<\/em> blogs. Let&#8217;s see this together.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">JMX Exporter vs Kafka Exporter<\/h2>\n\n\n\n<p>As Kafka is a Java based application, it would have been possible to monitor it with <a href=\"https:\/\/github.com\/prometheus\/jmx_exporter\" target=\"_blank\" rel=\"noreferrer noopener\">JMX Exporter<\/a> as I did for <a href=\"https:\/\/www.dbi-services.com\/blog\/apache-tomcat-monitoring-in-prometheus\/\" target=\"_blank\" rel=\"noreferrer noopener\">Apache Tomcat<\/a>. But this time, I will use an exporter that is accessing Kafka API.<\/p>\n\n\n\n<p>I am using this <a href=\"https:\/\/github.com\/danielqsj\/kafka_exporter\" target=\"_blank\" rel=\"noreferrer noopener\">Kafka Exporter<\/a> for demonstration purpose only.<\/p>\n\n\n\n<p>If you don&#8217;t want to alter your Kafka setup, it is possible to start this exporter in a docker container. Command is as easy as this (where kafka is the kafka server hostname):<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n$ docker run -d --rm -p 9308:9308 danielqsj\/kafka-exporter \\\n         --kafka.server=kafka:9092\n<\/pre><\/div>\n\n\n<p>Exporter will listen on port 9308 and log following lines:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nI0418 06:38:02.503148       1 kafka_exporter.go:792] Starting kafka_exporter (version=1.6.0, branch=master, revision=9d9cd654ca57e4f153d0d0b00ce36069b6a677c1)\nI0418 06:38:02.550605       1 kafka_exporter.go:963] Listening on HTTP :9308\n<\/pre><\/div>\n\n\n<p>I can test the access to the exposed port with a browser on http:\/\/kafka:9308\/metrics:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\n# HELP go_cgo_go_to_c_calls_calls_total Count of calls made from Go to C by the current process.\n# TYPE go_cgo_go_to_c_calls_calls_total counter\ngo_cgo_go_to_c_calls_calls_total 0\n# HELP go_gc_cycles_automatic_gc_cycles_total Count of completed GC cycles generated by the Go runtime.\n# TYPE go_gc_cycles_automatic_gc_cycles_total counter\ngo_gc_cycles_automatic_gc_cycles_total 7\n# HELP go_gc_cycles_forced_gc_cycles_total Count of completed GC cycles forced by the application.\n# TYPE go_gc_cycles_forced_gc_cycles_total counter\ngo_gc_cycles_forced_gc_cycles_total 0\n# HELP go_gc_cycles_total_gc_cycles_total Count of all completed GC cycles.\n# TYPE go_gc_cycles_total_gc_cycles_total counter\ngo_gc_cycles_total_gc_cycles_total 7\n# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.\n# TYPE go_gc_duration_seconds summary\ngo_gc_duration_seconds{quantile=&quot;0&quot;} 3.2e-05\ngo_gc_duration_seconds{quantile=&quot;0.25&quot;} 6.23e-05\ngo_gc_duration_seconds{quantile=&quot;0.5&quot;} 0.0001046\ngo_gc_duration_seconds{quantile=&quot;0.75&quot;} 0.0001833\ngo_gc_duration_seconds{quantile=&quot;1&quot;} 0.0002453\ngo_gc_duration_seconds_sum 0.0008797\ngo_gc_duration_seconds_count 7\n# HELP go_gc_heap_allocs_by_size_bytes_total Distribution of heap allocations by approximate size. Note that this does not include tiny objects as defined by \/gc\/heap\/tiny\/allocs:objects, only tiny blocks.\n# TYPE go_gc_heap_allocs_by_size_bytes_total histogram\ngo_gc_heap_allocs_by_size_bytes_total_bucket{le=&quot;8.999999999999998&quot;} 1120\ngo_gc_heap_allocs_by_size_bytes_total_bucket{le=&quot;24.999999999999996&quot;} 39780\ngo_gc_heap_allocs_by_size_bytes_total_bucket{le=&quot;64.99999999999999&quot;} 66615\ngo_gc_heap_allocs_by_size_bytes_total_bucket{le=&quot;144.99999999999997&quot;} 79645\ngo_gc_heap_allocs_by_size_bytes_total_bucket{le=&quot;320.99999999999994&quot;} 82112\n...\n<\/pre><\/div>\n\n\n<p>Everything looks good, but I don&#8217;t have anything interesting to monitor yet.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Kafka Topic<\/h2>\n\n\n\n<p>Kafka stores events (aka. record or messages) in topics. I will create one named MyTopic with that command:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n$ bin\/kafka-topics.sh --create --topic MyTopic --bootstrap-server localhost:9092\nCreated topic MyTopic.\n<\/pre><\/div>\n\n\n<p>To write an event into that topic, the command will look like:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n$ bin\/kafka-console-producer.sh --topic MyTopic --bootstrap-server localhost:9092\n<\/pre><\/div>\n\n\n<p>When producer will be started, you will be prompted with a &#8220;&gt;&#8221; which means it is ready to process. Each event will be a line of text as pressing Enter will send the message into the queue.<\/p>\n\n\n\n<p>To demonstrate that, I will open a producer and a consumer side by side:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png\" alt=\"\" class=\"wp-image-24642\" width=\"840\" height=\"50\" srcset=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png 1024w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-300x18.png 300w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-768x46.png 768w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1536x92.png 1536w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-2048x122.png 2048w\" sizes=\"auto, (max-width: 840px) 100vw, 840px\" \/><figcaption class=\"wp-element-caption\">Kafka producer and consumer<\/figcaption><\/figure>\n\n\n\n<p>Of course, events are rarely simple text string. It is usually structured data in Yaml or Json.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Prometheus Scraping<\/h2>\n\n\n\n<p>I have to modify Prometheus configuration file for it to pull metrics from exporter.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: yaml; title: ; notranslate\" title=\"\">\nscrape_configs:\n  - job_name: &#039;metrics&#039;\n    static_configs:\n    - targets: &#x5B;&#039;x.y.z.t:9308&#039;]\n      labels:\n        middleware_type: &#039;kafka&#039;\n<\/pre><\/div>\n\n\n<p>After triggering a configuration reload, I must check the target state. On Prometheus URL (http:\/\/a.b.c.d:9090\/), go to <em>Status <\/em>and then <em>Targets <\/em>menu:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"204\" src=\"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-08_45_14-Prometheus-Time-Series-Collection-and-Processing-Server-1-1024x204.png\" alt=\"\" class=\"wp-image-24639\" srcset=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-08_45_14-Prometheus-Time-Series-Collection-and-Processing-Server-1-1024x204.png 1024w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-08_45_14-Prometheus-Time-Series-Collection-and-Processing-Server-1-300x60.png 300w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-08_45_14-Prometheus-Time-Series-Collection-and-Processing-Server-1-768x153.png 768w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-08_45_14-Prometheus-Time-Series-Collection-and-Processing-Server-1-1536x306.png 1536w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-08_45_14-Prometheus-Time-Series-Collection-and-Processing-Server-1-2048x408.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">Prometheus Targets View<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Grafana<\/h2>\n\n\n\n<p>I am creating a new dashboard with following metric:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nkafka_topic_partition_current_offset{topic=&quot;MyTopic&quot;}\n<\/pre><\/div>\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"268\" src=\"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_45_58-Edit-panel-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox-1024x268.png\" alt=\"\" class=\"wp-image-24643\" srcset=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_45_58-Edit-panel-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox-1024x268.png 1024w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_45_58-Edit-panel-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox-300x79.png 300w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_45_58-Edit-panel-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox-768x201.png 768w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_45_58-Edit-panel-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox.png 1493w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">Grafana Dashboard<\/figcaption><\/figure>\n\n\n\n<p>As this offset keeps increasing, it is not relevant to draw it directly. Thus, I will use &#8220;<a href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/functions\/#rate\" target=\"_blank\" rel=\"noreferrer noopener\">rate<\/a>&#8221; function from Prometheus:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nrate(kafka_topic_partition_current_offset{topic=&quot;MyTopic&quot;}&#x5B;1m])\n<\/pre><\/div>\n\n\n<h2 class=\"wp-block-heading\">Kafka Producer Performance Test<\/h2>\n\n\n\n<p>Kafka includes a script to produce messages with several options. Command:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nbin\/kafka-producer-perf-test.sh --topic MyTopic --throughput 2000000 --num-records 2000000 --record-size 10 --producer-props bootstrap.servers=localhost:9092\n<\/pre><\/div>\n\n\n<p>Output gives a regular update on performances plus an additional summary at the end with percentiles:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\n105234 records sent, 21030.0 records\/sec (0.20 MB\/sec), 1233.0 ms avg latency, 2523.0 ms max latency.\n382135 records sent, 74014.1 records\/sec (0.71 MB\/sec), 2923.3 ms avg latency, 4572.0 ms max latency.\n1275372 records sent, 254921.4 records\/sec (2.43 MB\/sec), 5247.0 ms avg latency, 6122.0 ms max latency.\n2000000 records sent, 125557.159897 records\/sec (1.20 MB\/sec), 4362.06 ms avg latency, 6122.00 ms max latency, 4667 ms 50th, 6001 ms 95th, 6078 ms 99th, 6121 ms 99.9th.\n<\/pre><\/div>\n\n\n<p>Consumer output:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\n...\nCreateTime:1681804578211        Offset:17790650 KKRWEIFTZU\nCreateTime:1681804578211        Offset:17790651 YMNIYEGITJ\nCreateTime:1681804578211        Offset:17790652 UMSLDSLBWY\nCreateTime:1681804578211        Offset:17790653 LBGMSWGARD\n...\n<\/pre><\/div>\n\n\n<p>And, finally, the dashboard will look like:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"912\" height=\"334\" src=\"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-10_47_16-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox.png\" alt=\"\" class=\"wp-image-24647\" srcset=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-10_47_16-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox.png 912w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-10_47_16-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox-300x110.png 300w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-10_47_16-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox-768x281.png 768w\" sizes=\"auto, (max-width: 912px) 100vw, 912px\" \/><figcaption class=\"wp-element-caption\">Grafana dashboard<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>Enabling monitoring on Kafka with Prometheus\/Grafana is not more difficult than others exporters I configured in my previous monitoring in Prometheus blogs. Let&#8217;s see this together. JMX Exporter vs Kafka Exporter As Kafka is a Java based application, it would have been possible to monitor it with JMX Exporter as I did for Apache Tomcat. [&hellip;]<\/p>\n","protected":false},"author":40,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[197],"tags":[2257,1103,143],"type_dbi":[],"class_list":["post-24533","post","type-post","status-publish","format-standard","hentry","category-application-integration-middleware","tag-grafana","tag-kafka","tag-monitoring"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.4) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Kafka Monitoring in Prometheus - dbi Blog<\/title>\n<meta name=\"description\" content=\"This blog describes how to setup a Kafka exporter to get metrics into Prometheus and Grafana.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Kafka Monitoring in Prometheus\" \/>\n<meta property=\"og:description\" content=\"This blog describes how to setup a Kafka exporter to get metrics into Prometheus and Grafana.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2023-04-18T08:52:51+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-09-11T08:18:47+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png\" \/>\n<meta name=\"author\" content=\"Middleware Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Middleware Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/\"},\"author\":{\"name\":\"Middleware Team\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/8d8563acfc6e604cce6507f45bac0ea1\"},\"headline\":\"Kafka Monitoring in Prometheus\",\"datePublished\":\"2023-04-18T08:52:51+00:00\",\"dateModified\":\"2024-09-11T08:18:47+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/\"},\"wordCount\":380,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/04\\\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png\",\"keywords\":[\"Grafana\",\"kafka\",\"Monitoring\"],\"articleSection\":[\"Application integration &amp; Middleware\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/\",\"name\":\"Kafka Monitoring in Prometheus - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#primaryimage\"},\"thumbnailUrl\":\"http:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/04\\\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png\",\"datePublished\":\"2023-04-18T08:52:51+00:00\",\"dateModified\":\"2024-09-11T08:18:47+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/8d8563acfc6e604cce6507f45bac0ea1\"},\"description\":\"This blog describes how to setup a Kafka exporter to get metrics into Prometheus and Grafana.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#primaryimage\",\"url\":\"http:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/04\\\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png\",\"contentUrl\":\"http:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/04\\\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Kafka Monitoring in Prometheus\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/8d8563acfc6e604cce6507f45bac0ea1\",\"name\":\"Middleware Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"caption\":\"Middleware Team\"},\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/author\\\/middleware-team\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Kafka Monitoring in Prometheus - dbi Blog","description":"This blog describes how to setup a Kafka exporter to get metrics into Prometheus and Grafana.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/","og_locale":"en_US","og_type":"article","og_title":"Kafka Monitoring in Prometheus","og_description":"This blog describes how to setup a Kafka exporter to get metrics into Prometheus and Grafana.","og_url":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/","og_site_name":"dbi Blog","article_published_time":"2023-04-18T08:52:51+00:00","article_modified_time":"2024-09-11T08:18:47+00:00","og_image":[{"url":"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png","type":"","width":"","height":""}],"author":"Middleware Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Middleware Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/"},"author":{"name":"Middleware Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1"},"headline":"Kafka Monitoring in Prometheus","datePublished":"2023-04-18T08:52:51+00:00","dateModified":"2024-09-11T08:18:47+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/"},"wordCount":380,"commentCount":0,"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#primaryimage"},"thumbnailUrl":"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png","keywords":["Grafana","kafka","Monitoring"],"articleSection":["Application integration &amp; Middleware"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/","url":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/","name":"Kafka Monitoring in Prometheus - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#primaryimage"},"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#primaryimage"},"thumbnailUrl":"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png","datePublished":"2023-04-18T08:52:51+00:00","dateModified":"2024-09-11T08:18:47+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1"},"description":"This blog describes how to setup a Kafka exporter to get metrics into Prometheus and Grafana.","breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#primaryimage","url":"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png","contentUrl":"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png"},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Kafka Monitoring in Prometheus"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1","name":"Middleware Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","caption":"Middleware Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/middleware-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/24533","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/40"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=24533"}],"version-history":[{"count":14,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/24533\/revisions"}],"predecessor-version":[{"id":24650,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/24533\/revisions\/24650"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=24533"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=24533"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=24533"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=24533"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}