\nfor i in {0..100000}; do echo $i; fallocate -l 1m file${i};sleep 1;done\n<\/pre><\/div>\n\n\nAfter almost an hour, I am seeing this when I plot node_filesystem_free_bytes{mountpoint=\"\/\",job=\"node\"}<\/code>:<\/p>\n\n\n\nOf course, such a linear pattern does not reflect what we are seeing in a more real scenario, but this will help to confirm predict_linear<\/code> function works.<\/p>\n\n\n\nNow, we can add another query in Grafana:<\/p>\n\n\n
\npredict_linear(node_filesystem_free_bytes{mountpoint="\/",job="node"}[1h],3600)\n<\/pre><\/div>\n\n\n1h<\/code> means we are gathering a range-vector data of 1 hour; thus it will be able to forecast 1 hour after current time. 3600<\/code> is the number of seconds after the current time to compute forecast value.<\/p>\n\n\n\nScreenshot taken at 13:45.<\/figcaption><\/figure>\n\n\n\nIn yellow, we can see the predicted value in 1 hour (3600 seconds). If we compare predicted value 1 hour ago (yes, it sounds stupid) and current value we can see it matches:<\/p>\n\n\n\nScreenshot taken at 13:50.<\/figcaption><\/figure>\n\n\n\nNow, to display future data, we simply need to modify time range To <\/strong>field:<\/p>\n\n\n\nWhich will show following graph:<\/p>\n\n\n\nScreenshot taken at 14:00.<\/figcaption><\/figure>\n\n\n\nIt shows that free space prediction is 110 GB in 1 hour. Let’s do some math to check:<\/p>\n\n\n\n
Current occupancy is 113 GB. The script is filling at the rate of 3,6 GB\/h (1 MB\/s => 60 MB\/min => 3600 MB\/h), so we are good.<\/p>\n\n\n\n
Going Further in Future<\/h2>\n\n\n\n That’s interesting, but how would knowing the occupancy rate one hour in the future help me anyhow?<\/p>\n\n\n\n
Let’s create another query on 1 day:<\/p>\n\n\n
\npredict_linear(node_filesystem_free_bytes{mountpoint="\/",job="node"}[24h],86400)\n<\/pre><\/div>\n\n\nScreenshot taken at 14:15.<\/figcaption><\/figure>\n\n\n\nIf I move the mouse over the Predict 1d<\/strong> curve, I will notice that at a point in time, it will be a negative value. As you know available disk space can’t be a negative value which means that if I do not take any preventive measure, the partition will be full on the next day around 23:00.<\/p>\n\n\n\nTurn this into an Alert<\/h2>\n\n\n\n The idea is to get alerted whenever the prediction in next 24 hours will reach 0. So, the alert expression uses previous query result is less than 0.<\/p>\n\n\n\n
Here is an example of alerting rule for space usage:<\/p>\n\n\n
\ngroups:\n- name: Node\n rules:\n - alert: PredictedOutOfSpace\n expr: predict_linear(node_filesystem_free_bytes{mountpoint="\/",job="node"}[24h],86400) < 0\n for: 5m\n labels:\n severity: warning\n annotations:\n summary: Root Partition will be out of space\n<\/pre><\/div>\n\n\nIt is possible to make this raised any number of days before free space gets to 0 by adjusting the two parameters.<\/p>\n\n\n\n
Conclusion<\/h2>\n\n\n\n With this kind of alerting, I will be warned before we reach full partition. In the example above, assuming we are friday, if I look at current usage (ie. 104GB free space), nothing to worry about. But if we look at the prediction, we know we will have a full partition during the week-end.<\/p>\n\n\n\n
If we want to avoid being disturbed outside office hours, we must think how prediction could help us. Even if it is not as powerfull as Saas monitoring solution (linear regression is basic deep learning), I am sure we can do even better.<\/p>\n","protected":false},"excerpt":{"rendered":"
As I am focusing on monitoring subject, I was wondering if it is possible to set alarm based on prediction instead of simple thresholds. Dynatrace has predictive anomaly detection feature, but in this blog, I want to test if on-premises solution could help on this. We will see what Prometheus is capable of. Alarm and […]<\/p>\n","protected":false},"author":40,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[197,1320],"tags":[2257,143,2964,2234],"type_dbi":[],"class_list":["post-25373","post","type-post","status-publish","format-standard","hentry","category-application-integration-middleware","category-devops","tag-grafana","tag-monitoring","tag-prediction","tag-prometheus"],"acf":[],"yoast_head":"\n
Predict the Future with Prometheus - dbi Blog<\/title>\n<meta name=\"description\" content=\"How Prometheus could help to set better events thresholds.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Predict the Future with Prometheus\" \/>\n<meta property=\"og:description\" content=\"How Prometheus could help to set better events thresholds.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2023-05-23T14:57:37+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-09-11T08:18:43+00:00\" \/>\n<meta name=\"author\" content=\"Middleware Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Middleware Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/\"},\"author\":{\"name\":\"Middleware Team\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1\"},\"headline\":\"Predict the Future with Prometheus\",\"datePublished\":\"2023-05-23T14:57:37+00:00\",\"dateModified\":\"2024-09-11T08:18:43+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/\"},\"wordCount\":622,\"commentCount\":0,\"keywords\":[\"Grafana\",\"Monitoring\",\"prediction\",\"Prometheus\"],\"articleSection\":[\"Application integration & Middleware\",\"DevOps\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/\",\"name\":\"Predict the Future with Prometheus - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2023-05-23T14:57:37+00:00\",\"dateModified\":\"2024-09-11T08:18:43+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1\"},\"description\":\"How Prometheus could help to set better events thresholds.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Predict the Future with Prometheus\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1\",\"name\":\"Middleware Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"caption\":\"Middleware Team\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/middleware-team\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Predict the Future with Prometheus - dbi Blog","description":"How Prometheus could help to set better events thresholds.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/","og_locale":"en_US","og_type":"article","og_title":"Predict the Future with Prometheus","og_description":"How Prometheus could help to set better events thresholds.","og_url":"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/","og_site_name":"dbi Blog","article_published_time":"2023-05-23T14:57:37+00:00","article_modified_time":"2024-09-11T08:18:43+00:00","author":"Middleware Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Middleware Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/"},"author":{"name":"Middleware Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1"},"headline":"Predict the Future with Prometheus","datePublished":"2023-05-23T14:57:37+00:00","dateModified":"2024-09-11T08:18:43+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/"},"wordCount":622,"commentCount":0,"keywords":["Grafana","Monitoring","prediction","Prometheus"],"articleSection":["Application integration & Middleware","DevOps"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/","url":"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/","name":"Predict the Future with Prometheus - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2023-05-23T14:57:37+00:00","dateModified":"2024-09-11T08:18:43+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1"},"description":"How Prometheus could help to set better events thresholds.","breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/predict-the-future-with-prometheus\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Predict the Future with Prometheus"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1","name":"Middleware Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","caption":"Middleware Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/middleware-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/25373","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/40"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=25373"}],"version-history":[{"count":15,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/25373\/revisions"}],"predecessor-version":[{"id":25664,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/25373\/revisions\/25664"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=25373"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=25373"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=25373"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=25373"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"\"](\"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/05\/2023-05-23-13_35_52-File-System-Usage-Dashboards-Grafana-\u2014-Mozilla-Firefox.png\")
<\/figure>\n\n\n\n![\"\"](\"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/05\/2023-05-23-13_45_15-File-System-Usage-Dashboards-Grafana-\u2014-Mozilla-Firefox-1024x476.png\")
![\"\"](\"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/05\/2023-05-23-13_50_34-File-System-Usage-Dashboards-Grafana-\u2014-Mozilla-Firefox.png\")
![\"\"](\"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/05\/2023-05-23-13_55_35-File-System-Usage-Dashboards-Grafana-\u2014-Mozilla-Firefox.png\")
<\/figure>\n\n\n\n![\"\"](\"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/05\/2023-05-23-14_00_23-File-System-Usage-Dashboards-Grafana-\u2014-Mozilla-Firefox-1024x477.png\")
![\"\"](\"http:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2023\/05\/2023-05-23-14_17_20-File-System-Usage-Dashboards-Grafana-\u2014-Mozilla-Firefox.png\")