{"id":24533,"date":"2023-04-18T10:52:51","date_gmt":"2023-04-18T08:52:51","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/?p=24533"},"modified":"2023-04-18T10:52:51","modified_gmt":"2023-04-18T08:52:51","slug":"kafka-monitoring-in-prometheus","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/","title":{"rendered":"Kafka Monitoring in Prometheus"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Enabling monitoring on Kafka with Prometheus\/Grafana is not more difficult than others exporters I configured in my previous <em>monitoring in Prometheus<\/em> blogs. Let&#8217;s see this together.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">JMX Exporter vs Kafka Exporter<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">As Kafka is a Java based application, it would have been possible to monitor it with <a href=\"https:\/\/github.com\/prometheus\/jmx_exporter\" target=\"_blank\" rel=\"noreferrer noopener\">JMX Exporter<\/a> as I did for <a href=\"https:\/\/www.dbi-services.com\/blog\/apache-tomcat-monitoring-in-prometheus\/\" target=\"_blank\" rel=\"noreferrer noopener\">Apache Tomcat<\/a>. But this time, I will use an exporter that is accessing Kafka API.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I am using this <a href=\"https:\/\/github.com\/danielqsj\/kafka_exporter\" target=\"_blank\" rel=\"noreferrer noopener\">Kafka Exporter<\/a> for demonstration purpose only.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you don&#8217;t want to alter your Kafka setup, it is possible to start this exporter in a docker container. Command is as easy as this (where kafka is the kafka server hostname):<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n$ docker run -d --rm -p 9308:9308 danielqsj\/kafka-exporter \\\n         --kafka.server=kafka:9092\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">Exporter will listen on port 9308 and log following lines:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nI0418 06:38:02.503148       1 kafka_exporter.go:792] Starting kafka_exporter (version=1.6.0, branch=master, revision=9d9cd654ca57e4f153d0d0b00ce36069b6a677c1)\nI0418 06:38:02.550605       1 kafka_exporter.go:963] Listening on HTTP :9308\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">I can test the access to the exposed port with a browser on http:\/\/kafka:9308\/metrics:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\n# HELP go_cgo_go_to_c_calls_calls_total Count of calls made from Go to C by the current process.\n# TYPE go_cgo_go_to_c_calls_calls_total counter\ngo_cgo_go_to_c_calls_calls_total 0\n# HELP go_gc_cycles_automatic_gc_cycles_total Count of completed GC cycles generated by the Go runtime.\n# TYPE go_gc_cycles_automatic_gc_cycles_total counter\ngo_gc_cycles_automatic_gc_cycles_total 7\n# HELP go_gc_cycles_forced_gc_cycles_total Count of completed GC cycles forced by the application.\n# TYPE go_gc_cycles_forced_gc_cycles_total counter\ngo_gc_cycles_forced_gc_cycles_total 0\n# HELP go_gc_cycles_total_gc_cycles_total Count of all completed GC cycles.\n# TYPE go_gc_cycles_total_gc_cycles_total counter\ngo_gc_cycles_total_gc_cycles_total 7\n# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.\n# TYPE go_gc_duration_seconds summary\ngo_gc_duration_seconds{quantile=&quot;0&quot;} 3.2e-05\ngo_gc_duration_seconds{quantile=&quot;0.25&quot;} 6.23e-05\ngo_gc_duration_seconds{quantile=&quot;0.5&quot;} 0.0001046\ngo_gc_duration_seconds{quantile=&quot;0.75&quot;} 0.0001833\ngo_gc_duration_seconds{quantile=&quot;1&quot;} 0.0002453\ngo_gc_duration_seconds_sum 0.0008797\ngo_gc_duration_seconds_count 7\n# HELP go_gc_heap_allocs_by_size_bytes_total Distribution of heap allocations by approximate size. Note that this does not include tiny objects as defined by \/gc\/heap\/tiny\/allocs:objects, only tiny blocks.\n# TYPE go_gc_heap_allocs_by_size_bytes_total histogram\ngo_gc_heap_allocs_by_size_bytes_total_bucket{le=&quot;8.999999999999998&quot;} 1120\ngo_gc_heap_allocs_by_size_bytes_total_bucket{le=&quot;24.999999999999996&quot;} 39780\ngo_gc_heap_allocs_by_size_bytes_total_bucket{le=&quot;64.99999999999999&quot;} 66615\ngo_gc_heap_allocs_by_size_bytes_total_bucket{le=&quot;144.99999999999997&quot;} 79645\ngo_gc_heap_allocs_by_size_bytes_total_bucket{le=&quot;320.99999999999994&quot;} 82112\n...\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">Everything looks good, but I don&#8217;t have anything interesting to monitor yet.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Kafka Topic<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka stores events (aka. record or messages) in topics. I will create one named MyTopic with that command:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n$ bin\/kafka-topics.sh --create --topic MyTopic --bootstrap-server localhost:9092\nCreated topic MyTopic.\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">To write an event into that topic, the command will look like:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n$ bin\/kafka-console-producer.sh --topic MyTopic --bootstrap-server localhost:9092\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">When producer will be started, you will be prompted with a &#8220;&gt;&#8221; which means it is ready to process. Each event will be a line of text as pressing Enter will send the message into the queue.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To demonstrate that, I will open a producer and a consumer side by side:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png\" alt=\"\" class=\"wp-image-24642\" width=\"840\" height=\"50\" srcset=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png 1024w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-300x18.png 300w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-768x46.png 768w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1536x92.png 1536w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-2048x122.png 2048w\" sizes=\"auto, (max-width: 840px) 100vw, 840px\" \/><figcaption class=\"wp-element-caption\">Kafka producer and consumer<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Of course, events are rarely simple text string. It is usually structured data in Yaml or Json.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Prometheus Scraping<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I have to modify Prometheus configuration file for it to pull metrics from exporter.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: yaml; title: ; notranslate\" title=\"\">\nscrape_configs:\n  - job_name: &#039;metrics&#039;\n    static_configs:\n    - targets: &#x5B;&#039;x.y.z.t:9308&#039;]\n      labels:\n        middleware_type: &#039;kafka&#039;\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">After triggering a configuration reload, I must check the target state. On Prometheus URL (http:\/\/a.b.c.d:9090\/), go to <em>Status <\/em>and then <em>Targets <\/em>menu:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"204\" src=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-08_45_14-Prometheus-Time-Series-Collection-and-Processing-Server-1-1024x204.png\" alt=\"\" class=\"wp-image-24639\" srcset=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-08_45_14-Prometheus-Time-Series-Collection-and-Processing-Server-1-1024x204.png 1024w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-08_45_14-Prometheus-Time-Series-Collection-and-Processing-Server-1-300x60.png 300w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-08_45_14-Prometheus-Time-Series-Collection-and-Processing-Server-1-768x153.png 768w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-08_45_14-Prometheus-Time-Series-Collection-and-Processing-Server-1-1536x306.png 1536w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-08_45_14-Prometheus-Time-Series-Collection-and-Processing-Server-1-2048x408.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">Prometheus Targets View<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Grafana<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">I am creating a new dashboard with following metric:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nkafka_topic_partition_current_offset{topic=&quot;MyTopic&quot;}\n<\/pre><\/div>\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"268\" src=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_45_58-Edit-panel-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox-1024x268.png\" alt=\"\" class=\"wp-image-24643\" srcset=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_45_58-Edit-panel-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox-1024x268.png 1024w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_45_58-Edit-panel-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox-300x79.png 300w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_45_58-Edit-panel-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox-768x201.png 768w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_45_58-Edit-panel-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox.png 1493w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">Grafana Dashboard<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">As this offset keeps increasing, it is not relevant to draw it directly. Thus, I will use &#8220;<a href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/functions\/#rate\" target=\"_blank\" rel=\"noreferrer noopener\">rate<\/a>&#8221; function from Prometheus:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nrate(kafka_topic_partition_current_offset{topic=&quot;MyTopic&quot;}&#x5B;1m])\n<\/pre><\/div>\n\n\n<h2 class=\"wp-block-heading\">Kafka Producer Performance Test<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Kafka includes a script to produce messages with several options. Command:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nbin\/kafka-producer-perf-test.sh --topic MyTopic --throughput 2000000 --num-records 2000000 --record-size 10 --producer-props bootstrap.servers=localhost:9092\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">Output gives a regular update on performances plus an additional summary at the end with percentiles:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\n105234 records sent, 21030.0 records\/sec (0.20 MB\/sec), 1233.0 ms avg latency, 2523.0 ms max latency.\n382135 records sent, 74014.1 records\/sec (0.71 MB\/sec), 2923.3 ms avg latency, 4572.0 ms max latency.\n1275372 records sent, 254921.4 records\/sec (2.43 MB\/sec), 5247.0 ms avg latency, 6122.0 ms max latency.\n2000000 records sent, 125557.159897 records\/sec (1.20 MB\/sec), 4362.06 ms avg latency, 6122.00 ms max latency, 4667 ms 50th, 6001 ms 95th, 6078 ms 99th, 6121 ms 99.9th.\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">Consumer output:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\n...\nCreateTime:1681804578211        Offset:17790650 KKRWEIFTZU\nCreateTime:1681804578211        Offset:17790651 YMNIYEGITJ\nCreateTime:1681804578211        Offset:17790652 UMSLDSLBWY\nCreateTime:1681804578211        Offset:17790653 LBGMSWGARD\n...\n<\/pre><\/div>\n\n\n<p class=\"wp-block-paragraph\">And, finally, the dashboard will look like:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"912\" height=\"334\" src=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-10_47_16-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox.png\" alt=\"\" class=\"wp-image-24647\" srcset=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-10_47_16-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox.png 912w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-10_47_16-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox-300x110.png 300w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-10_47_16-New-dashboard-Dashboards-Grafana-\u2014-Mozilla-Firefox-768x281.png 768w\" sizes=\"auto, (max-width: 912px) 100vw, 912px\" \/><figcaption class=\"wp-element-caption\">Grafana dashboard<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>Enabling monitoring on Kafka with Prometheus\/Grafana is not more difficult than others exporters I configured in my previous monitoring in Prometheus blogs. Let&#8217;s see this together. JMX Exporter vs Kafka Exporter As Kafka is a Java based application, it would have been possible to monitor it with JMX Exporter as I did for Apache Tomcat.&hellip;&nbsp;<\/p>\n","protected":false},"author":40,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"","neve_meta_content_width":0,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":"","neve_meta_reading_time":"","_themeisle_gutenberg_block_has_review":false,"footnotes":""},"categories":[4134],"tags":[2257,1103,143],"class_list":["post-24533","post","type-post","status-publish","format-standard","category-application-server","tag-grafana","tag-kafka","tag-monitoring"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v28.8 (Yoast SEO v28.9) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Kafka Monitoring in Prometheus - dbi services Blog<\/title>\n<meta name=\"description\" content=\"This blog describes how to setup a Kafka exporter to get metrics into Prometheus and Grafana.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Kafka Monitoring in Prometheus\" \/>\n<meta property=\"og:description\" content=\"This blog describes how to setup a Kafka exporter to get metrics into Prometheus and Grafana.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi services Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/dbiservices\/\" \/>\n<meta property=\"article:published_time\" content=\"2023-04-18T08:52:51+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet.png\" \/>\n\t<meta property=\"og:image:width\" content=\"3818\" \/>\n\t<meta property=\"og:image:height\" content=\"228\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Middleware Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@dbiservices\" \/>\n<meta name=\"twitter:site\" content=\"@dbiservices\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Middleware Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/\"},\"author\":{\"name\":\"Middleware Team\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/8a0bfec3c3bd1f27d3d452eaa592bfb8\"},\"headline\":\"Kafka Monitoring in Prometheus\",\"datePublished\":\"2023-04-18T08:52:51+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/\"},\"wordCount\":380,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/04\\\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png\",\"keywords\":[\"Grafana\",\"kafka\",\"Monitoring\"],\"articleSection\":[\"Application Server\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/\",\"name\":\"Kafka Monitoring in Prometheus - dbi services Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/04\\\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png\",\"datePublished\":\"2023-04-18T08:52:51+00:00\",\"description\":\"This blog describes how to setup a Kafka exporter to get metrics into Prometheus and Grafana.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/04\\\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet.png\",\"contentUrl\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/04\\\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet.png\",\"width\":3818,\"height\":228},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/kafka-monitoring-in-prometheus\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Kafka Monitoring in Prometheus\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\",\"name\":\"dbi services Blog\",\"description\":\"#BeSharing\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/#organization\",\"name\":\"dbi services\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/dbi-services-logo.png\",\"contentUrl\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/dbi-services-logo.png\",\"width\":512,\"height\":512,\"caption\":\"dbi services\"},\"image\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/dbiservices\\\/\",\"https:\\\/\\\/x.com\\\/dbiservices\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/dbi-services\\\/\",\"https:\\\/\\\/www.youtube.com\\\/user\\\/dbiservices\"],\"description\":\"Swiss IT consulting and services company specialized in efficient data infrastructures and platforms: databases, application servers, cloud, DevOps, digitalization, analytics, data quality and monitoring. Headquartered in Del\u00e9mont with offices in Nyon, Basel, Bern and Zurich.\",\"email\":\"info@dbi-services.com\",\"telephone\":\"+41 32 422 96 00\",\"legalName\":\"dbi services SA\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/8a0bfec3c3bd1f27d3d452eaa592bfb8\",\"name\":\"Middleware Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"caption\":\"Middleware Team\"},\"description\":\"Posts published under the Middleware Team account are collective contributions by the dbi services middleware and enterprise content management consultants. The practice covers ECM platforms such as OpenText Documentum, Alfresco, and M-Files, as well as application servers and integration technologies including WebLogic, JBoss, Tomcat, and the Elastic Stack, from architecture and deployment to operations under service level agreements.\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/author\\\/middleware-team\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Kafka Monitoring in Prometheus - dbi services Blog","description":"This blog describes how to setup a Kafka exporter to get metrics into Prometheus and Grafana.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/","og_locale":"en_US","og_type":"article","og_title":"Kafka Monitoring in Prometheus","og_description":"This blog describes how to setup a Kafka exporter to get metrics into Prometheus and Grafana.","og_url":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/","og_site_name":"dbi services Blog","article_publisher":"https:\/\/www.facebook.com\/dbiservices\/","article_published_time":"2023-04-18T08:52:51+00:00","og_image":[{"width":3818,"height":228,"url":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet.png","type":"image\/png"}],"author":"Middleware Team","twitter_card":"summary_large_image","twitter_creator":"@dbiservices","twitter_site":"@dbiservices","twitter_misc":{"Written by":"Middleware Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/"},"author":{"name":"Middleware Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8a0bfec3c3bd1f27d3d452eaa592bfb8"},"headline":"Kafka Monitoring in Prometheus","datePublished":"2023-04-18T08:52:51+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/"},"wordCount":380,"commentCount":0,"publisher":{"@id":"https:\/\/www.dbi-services.com\/#organization"},"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#primaryimage"},"thumbnailUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png","keywords":["Grafana","kafka","Monitoring"],"articleSection":["Application Server"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/","url":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/","name":"Kafka Monitoring in Prometheus - dbi services Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#primaryimage"},"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#primaryimage"},"thumbnailUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet-1024x61.png","datePublished":"2023-04-18T08:52:51+00:00","description":"This blog describes how to setup a Kafka exporter to get metrics into Prometheus and Grafana.","breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#primaryimage","url":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet.png","contentUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2023\/04\/2023-04-18-09_15_20-Formatting-The-Apache-Kafka-Console-Consumer-Output-_-by-Ivelina-Yordanova-_-Bet.png","width":3818,"height":228},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/kafka-monitoring-in-prometheus\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Kafka Monitoring in Prometheus"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi services Blog","description":"#BeSharing","publisher":{"@id":"https:\/\/www.dbi-services.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.dbi-services.com\/#organization","name":"dbi services","url":"https:\/\/www.dbi-services.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2026\/09\/dbi-services-logo.png","contentUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/2026\/09\/dbi-services-logo.png","width":512,"height":512,"caption":"dbi services"},"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/dbiservices\/","https:\/\/x.com\/dbiservices","https:\/\/www.linkedin.com\/company\/dbi-services\/","https:\/\/www.youtube.com\/user\/dbiservices"],"description":"Swiss IT consulting and services company specialized in efficient data infrastructures and platforms: databases, application servers, cloud, DevOps, digitalization, analytics, data quality and monitoring. Headquartered in Del\u00e9mont with offices in Nyon, Basel, Bern and Zurich.","email":"info@dbi-services.com","telephone":"+41 32 422 96 00","legalName":"dbi services SA"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8a0bfec3c3bd1f27d3d452eaa592bfb8","name":"Middleware Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","caption":"Middleware Team"},"description":"Posts published under the Middleware Team account are collective contributions by the dbi services middleware and enterprise content management consultants. The practice covers ECM platforms such as OpenText Documentum, Alfresco, and M-Files, as well as application servers and integration technologies including WebLogic, JBoss, Tomcat, and the Elastic Stack, from architecture and deployment to operations under service level agreements.","url":"https:\/\/www.dbi-services.com\/blog\/author\/middleware-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/24533","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/40"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=24533"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/24533\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=24533"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=24533"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=24533"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}