{"id":25453,"date":"2023-05-25T18:18:09","date_gmt":"2023-05-25T16:18:09","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/?p=25453"},"modified":"2024-09-11T10:18:42","modified_gmt":"2024-09-11T08:18:42","slug":"monitoring-short-living-processes-with-prometheus","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/","title":{"rendered":"Monitoring Short Living Processes with Prometheus"},"content":{"rendered":"\n<p>With the way Prometheus is gathering metrics, pull, it is not possible to get metrics from short living processes like batch jobs as:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>batch jobs can be shorter than <code>scrap_interval<\/code><\/li>\n\n\n\n<li>batch timings and <code>scrap_interval<\/code> are not aligned<\/li>\n<\/ul>\n\n\n\n<p>In this blog, I will explain how to bypass this limitation with two possibles solutions: The Textfile Collector and push gateway.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Textfile Collector<\/h2>\n\n\n\n<p>Promotheus <a href=\"https:\/\/github.com\/prometheus\/node_exporter\" target=\"_blank\" rel=\"noreferrer noopener\">node_exporter<\/a> can parse files and expose results on export page. For that, we need to indicate to node_exporter which directory to parse by adding <code>--collector.textfile.directory<\/code> at startup command. For example:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n.\/node_exporter --collector.textfile.directory \/root\/batch_results\/\n<\/pre><\/div>\n\n\n<p>node_exporter will parse all files with <code>.prom<\/code> extension. To avoid any access concurrency issue to the file, our batch script must write result in a temporary file and move it to a <code>.prom<\/code> file at very last step.<\/p>\n\n\n\n<p> To start simple, I will just write current timestamp in the file <code>my_script_end_time.prom<\/code> ended with a uniq number like process id (ie. $$):<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\necho my_script_end_time $(date +%s) &gt; my_script_end_time.prom.$$\n<\/pre><\/div>\n\n\n<p>This will create a file named <code>my_script_end_time.prom.36636<\/code> with following content:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nmy_script_end_time 1685025635\n<\/pre><\/div>\n\n\n<p>Next, for node exporter to parse it, I must remove the extra dot and PID with:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\nmv my_script_end_time.prom.36636 my_script_end_time.prom\n<\/pre><\/div>\n\n\n<p>As soon as I do that, timing will be visible on node exporter (ie. http:\/\/server:9100):<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\n$ curl -s &quot;http:\/\/localhost:9100\/metrics&quot; | grep -n my_script\n109:# HELP my_script_end_time Metric read from \/root\/batch_results\/my_script_end_time.prom\n110:# TYPE my_script_end_time untyped\n111:my_script_end_time 1.685025635e+09\n1041:node_textfile_mtime_seconds{file=&quot;\/root\/batch_results\/my_script_end_time.prom&quot;} 1.685025635e+09\n<\/pre><\/div>\n\n\n<p>Note that:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The last line indicates which file has been parsed and when it was last modified.<\/li>\n\n\n\n<li>Metric has no type. This could be changed by adding a line before (ie. <code># TYPE my_script_end_time counter<\/code>)<\/li>\n<\/ul>\n\n\n\n<p>With such a simple method, it is easy to instrument any script (with any scripting language) as it is just a writing into a file.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Push Gateway<\/h2>\n\n\n\n<p><a href=\"https:\/\/prometheus.io\/docs\/practices\/pushing\/\" target=\"_blank\" rel=\"noreferrer noopener\">Prometheus documentation<\/a> state that there is limitation on using a gateway and one valid use case:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p>Usually, the only valid use case for the Pushgateway is for capturing the outcome of a service-level batch job.<\/p>\n<cite>Promotheus documentation: When to use the push gateway<\/cite><\/blockquote>\n\n\n\n<p>Push gateway is a process which is somehow caching data for scrapper to have time to get metrics, thus it is as easy as adding any other Prometheus target. Next, we can try to &#8220;push&#8221; data into the gateway with a curl command:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\ncat &lt;&lt;EOF | curl --data-binary @- http:\/\/&lt;pushGW&gt;:9091\/metrics\/job\/my_job\/instance\/my_instance\n# TYPE my_metric gauge\n# HELP my_metric Processed Records\nmy_metric 12345\nEOF\n<\/pre><\/div>\n\n\n<p>Empty feedback and going back to prompt means it is ok. We can verify immediately with a curl command as well:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: bash; title: ; notranslate\" title=\"\">\n$ curl -s http:\/\/&lt;pushGW&gt;:9091\/metrics | grep my_\n# HELP my_metric Processed Records\n# TYPE my_metric gauge\nmy_metric{instance=&quot;my_instance&quot;,job=&quot;my_job&quot;} 12345\npush_failure_time_seconds{instance=&quot;my_instance&quot;,job=&quot;my_job&quot;} 0\npush_time_seconds{instance=&quot;my_instance&quot;,job=&quot;my_job&quot;} 1.6850300553978374e+09\n<\/pre><\/div>\n\n\n<p><code>job<\/code> and <code>instance<\/code> labels are reserved labels and thus will be renamed by Prometheus to <code>exported_job<\/code> and <code>exported_instance<\/code>. Prometheus will set instance to the push gateway host and port which is not convenient when push gateway is used by multiple hosts and\/or batches. To preserve <code>instance<\/code> and <code>job <\/code>labels, we must add a parameter to <code>prometheus.yml<\/code>:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: yaml; title: ; notranslate\" title=\"\">\n  - job_name: &#039;pushgateway&#039;\n    static_configs:\n    - targets: &#x5B;&#039;&lt;pushGW&gt;:9091&#039;]\n    honor_labels: true\n<\/pre><\/div>\n\n\n<p>Finally, I can find the result in Prometheus:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"865\" height=\"298\" src=\"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/05\/2023-05-25-18_02_28-Prometheus-Time-Series-Collection-and-Processing-Server-\u2014-Mozilla-Firefox.png\" alt=\"\" class=\"wp-image-25467\" srcset=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/05\/2023-05-25-18_02_28-Prometheus-Time-Series-Collection-and-Processing-Server-\u2014-Mozilla-Firefox.png 865w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/05\/2023-05-25-18_02_28-Prometheus-Time-Series-Collection-and-Processing-Server-\u2014-Mozilla-Firefox-300x103.png 300w, https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/05\/2023-05-25-18_02_28-Prometheus-Time-Series-Collection-and-Processing-Server-\u2014-Mozilla-Firefox-768x265.png 768w\" sizes=\"auto, (max-width: 865px) 100vw, 865px\" \/><figcaption class=\"wp-element-caption\">Prometheus GUI<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>With the way Prometheus is gathering metrics, pull, it is not possible to get metrics from short living processes like batch jobs as: In this blog, I will explain how to bypass this limitation with two possibles solutions: The Textfile Collector and push gateway. Textfile Collector Promotheus node_exporter can parse files and expose results on [&hellip;]<\/p>\n","protected":false},"author":40,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[197,1320],"tags":[2965,2234],"type_dbi":[],"class_list":["post-25453","post","type-post","status-publish","format-standard","hentry","category-application-integration-middleware","category-devops","tag-batch","tag-prometheus"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Monitoring Short Living Processes with Prometheus - dbi Blog<\/title>\n<meta name=\"description\" content=\"What are the possibilities for monitoring short living processes with Prometheus ?\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Monitoring Short Living Processes with Prometheus\" \/>\n<meta property=\"og:description\" content=\"What are the possibilities for monitoring short living processes with Prometheus ?\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2023-05-25T16:18:09+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-09-11T08:18:42+00:00\" \/>\n<meta name=\"author\" content=\"Middleware Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Middleware Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/\"},\"author\":{\"name\":\"Middleware Team\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1\"},\"headline\":\"Monitoring Short Living Processes with Prometheus\",\"datePublished\":\"2023-05-25T16:18:09+00:00\",\"dateModified\":\"2024-09-11T08:18:42+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/\"},\"wordCount\":420,\"commentCount\":0,\"keywords\":[\"Batch\",\"Prometheus\"],\"articleSection\":[\"Application integration &amp; Middleware\",\"DevOps\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/\",\"name\":\"Monitoring Short Living Processes with Prometheus - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2023-05-25T16:18:09+00:00\",\"dateModified\":\"2024-09-11T08:18:42+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1\"},\"description\":\"What are the possibilities for monitoring short living processes with Prometheus ?\",\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Monitoring Short Living Processes with Prometheus\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1\",\"name\":\"Middleware Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"caption\":\"Middleware Team\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/middleware-team\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Monitoring Short Living Processes with Prometheus - dbi Blog","description":"What are the possibilities for monitoring short living processes with Prometheus ?","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/","og_locale":"en_US","og_type":"article","og_title":"Monitoring Short Living Processes with Prometheus","og_description":"What are the possibilities for monitoring short living processes with Prometheus ?","og_url":"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/","og_site_name":"dbi Blog","article_published_time":"2023-05-25T16:18:09+00:00","article_modified_time":"2024-09-11T08:18:42+00:00","author":"Middleware Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Middleware Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/"},"author":{"name":"Middleware Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1"},"headline":"Monitoring Short Living Processes with Prometheus","datePublished":"2023-05-25T16:18:09+00:00","dateModified":"2024-09-11T08:18:42+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/"},"wordCount":420,"commentCount":0,"keywords":["Batch","Prometheus"],"articleSection":["Application integration &amp; Middleware","DevOps"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/","url":"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/","name":"Monitoring Short Living Processes with Prometheus - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2023-05-25T16:18:09+00:00","dateModified":"2024-09-11T08:18:42+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1"},"description":"What are the possibilities for monitoring short living processes with Prometheus ?","breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/monitoring-short-living-processes-with-prometheus\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Monitoring Short Living Processes with Prometheus"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1","name":"Middleware Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","caption":"Middleware Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/middleware-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/25453","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/40"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=25453"}],"version-history":[{"count":22,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/25453\/revisions"}],"predecessor-version":[{"id":25476,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/25453\/revisions\/25476"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=25453"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=25453"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=25453"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=25453"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}