Prometheus is a popular open-source monitoring system that helps track and alert for various applications. However, regarding security, there are some essential things to consider.
Prometheus server, for instance, does not provide authentication or TLS encryption out of the box, meaning anyone accessing the server’s HTTP endpoints can access all of our time-series data.<\/p>\n\n\n\n
So, what solutions are available?
One solution would be to secure the Prometheus server accesses using a reverse proxy, encrypting the communications between clients and a server.
We can implement this security layer with an Apache or Nginx server configured as a reverse proxy that implements TLS encryption, authentication and other security features.<\/p>\n\n\n\n
When it comes to exporters, many of them do provide some security features such as authentication and\/or TLS encryption, and we can set up your Prometheus server configuration to interact with those secure exporters.
This blog post will focus on securing communications between exporters and the Prometheus server by implementing authentication and TLS encryption. I will dedicate a separate blog post for securing the Prometheus server accesses using a reverse proxy like NGINX.<\/p>\n\n\n\n
Let’s assume we have a configuration with a Prometheus server that monitors one server with the Node Exporter configured as a service with minimal configuration. The idea is first to implement authentication and then TLS encryption.<\/p>\n\n\n\n
To do this, we will create a configuration file containing our exporter’s encrypted credentials.
<\/p>\n\n\n\n
First, log in to our node01, and install the apache2-utils package, which we will use to create a hash for the password.<\/p>\n\n\n
\napt update\napt install apache2-utils -y\n<\/pre><\/div>\n\n\nGenerate the password hash<\/p>\n\n\n
\nhtpasswd -nBC 10 "" | tr -d ':\\n'; echo\n<\/pre><\/div>\n\n\nThe system will prompt us to enter our password twice, as shown below:<\/p>\n\n\n\n
New password:\nRe-type new password:<\/code><\/pre>\n\n\n\nAfterwards, a hashed value of our password will be provided. We can then proceed to edit the configuration file located at \/etc\/node_exporter\/config.yml<\/code> and add the lines below (of course, replace the placeholder by the hashed password generated in the previous step):<\/p>\n\n\n\nbasic_auth_users:\n prometheus: ###our-hashed-password###\n\n<\/pre><\/div>\n\n\nWith the node_exporter configuration file created, it is necessary to declare it in the node_exporter systemd service file so that it is considered.
For this, we have to edit the \/etc\/systemd\/system\/node_exporter.service<\/code> and add the argument --web.config<\/code> in the ExecStart<\/strong> line, as below:<\/p>\n\n\n\n[Unit]\nDescription=Node Exporter\nAfter=network.target\n\n[Service]\nUser=nodexporter\nGroup=nodexporter\nType=simple\nExecStart=\/usr\/local\/bin\/node_exporter --web.config=\/etc\/node_exporter\/config.yml\n\n[Install]\nWantedBy=multi-user.target\n<\/pre><\/div>\n\n\nReload the daemon and Restart the node_exporter service:<\/p>\n\n\n
\nsystemctl daemon-reload\nsystemctl restart node_exporter\n<\/pre><\/div>\n\n\nWe can confirm the modifications using the “curl” command to access the metrics curl http:\/\/dbinode01:9100\/metrics<\/code>. The output should return Unauthorize<\/mark>.
We must provide the username and password in our curl command to access the metrics.<\/p>\n\n\n\ncurl -u prometheus:<clear-password> http:\/\/dbinode01:9100\/metrics\n<\/pre><\/div>\n\n\nNow that authentication has been implemented at the exporter level, and we need to configure it at the Prometheus server level.
Without this, the job that scrapes metrics from our node_exporter will return a “401 Unauthorized” error. To do this, log in to our Prometheus server, edit the configuration file, and add the following configuration to the job that is supposed to collect metrics from our node exporter (which we will refer to as “nodes” here).<\/p>\n\n\n\n
On our Prometheus server, edit the Prometheus configuration file<\/p>\n\n\n
\nvi \/etc\/prometheus\/prometheus.yml\n<\/pre><\/div>\n\n\nAdd the following line under - job_name: \"nodes\"<\/code><\/p>\n\n\n\nbasic_auth:\n username: prometheus\n password: clear-password\n<\/pre><\/div>\n\n\nRestart our Prometheus service:<\/p>\n\n\n
\nsystemctl restart prometheus\n<\/pre><\/div>\n\n\nTLS encryption<\/h3>\n\n\n\n
With authentication in place, we will now focus on encrypting the communication between the Prometheus server and its exporter.<\/p>\n\n\n\n
Let’s start by generating a certificate and its key on our node01 by running the following command:<\/p>\n\n\n
\nopenssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout node_exporter.key -out node_exporter.crt -subj "\/C=CH\/ST=Switzerland\/L=Basel\/O=dbi\/CN=localhost" -addext "subjectAltName = DNS:localhost"\n<\/pre><\/div>\n\n\nMove the crt<\/mark> and the key file<\/mark> under the node_exporter folder \/etc\/node_exporter\/<\/code><\/p>\n\n\n\nmv node_exporter.crt node_exporter.key \/etc\/node_exporter\/\n<\/pre><\/div>\n\n\nChange the ownership:<\/p>\n\n\n
\nchown nodexporter.nodexporter \/etc\/node_exporter\/node_exporter.key\nchown nodexporter.nodexporter \/etc\/node_exporter\/node_exporter.crt\n<\/pre><\/div>\n\n\nEdit node_exporter configuration file \/etc\/node_exporter\/config.yml<\/code> and add the following block:<\/p>\n\n\n\ntls_server_config:\n cert_file: node_exporter.crt\n key_file: node_exporter.key\n<\/pre><\/div>\n\n\nRestart the node_exporter service to load the new configuration<\/p>\n\n\n
\nsystemctl restart node_exporter\n<\/pre><\/div>\n\n\nWe can confirm our modifications by using the “curl” command with -k<\/code> flag to allow connections to insecure HTTPS servers. <\/p>\n\n\n\ncurl -u prometheus:<clear-password> -k https:\/\/dbinode01:9100\/metrics\n<\/pre><\/div>\n\n\nThe finish line is in sight, and we need to configure the Prometheus server to use HTTPS for scraping the node_exporter.
To do this, we need to retrieve the certificate and its key, place them in the Prometheus directory, and update the scraping job configuration in the Prometheus configuration to allow TLS communication.
Let’s copy the certificates to our Prometheus server and change the ownership.<\/p>\n\n\n
\nscp root@dbinode01:\/etc\/node_exporter\/node_exporter.crt \/etc\/prometheus\/node_exporter.crt\nchown prometheus:prometheus \/etc\/prometheus\/node_exporter.crt\n<\/pre><\/div>\n\n\nEdit \/etc\/prometheus\/prometheus.yml<\/code> file and add the lines provided below under - job_name: \"nodes\"<\/code><\/p>\n\n\n\n scheme: https\n tls_config:\n ca_file: \/etc\/prometheus\/node_exporter.crt\n insecure_skip_verify:true\n<\/pre><\/div>\n\n\nRestart Prometheus service<\/p>\n\n\n
\nsystemctl restart prometheus\n<\/pre><\/div>\n\n\nAnd that’s it.<\/p>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
That’s how we can establish authentication and encryption between our exporters and the Prometheus server.<\/p>\n\n\n\n
It’s crucial to take security seriously and carefully consider the security implications of our Prometheus ecosystem.<\/p>\n","protected":false},"excerpt":{"rendered":"
Prometheus is a popular open-source monitoring system that helps track and alert for various applications. However, regarding security, there are some essential things to consider. Prometheus server, for instance, does not provide authentication or TLS encryption out of the box, meaning anyone accessing the server’s HTTP endpoints can access all of our time-series data. So, […]<\/p>\n","protected":false},"author":40,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[229,1320],"tags":[995,143,2234,2321,2211],"type_dbi":[],"class_list":["post-23653","post","type-post","status-publish","format-standard","hentry","category-database-administration-monitoring","category-devops","tag-authentication","tag-monitoring","tag-prometheus","tag-secure","tag-tls"],"acf":[],"yoast_head":"\n
How to keep your Prometheus ecosystem secure - dbi Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/how-to-keep-your-prometheus-ecosystem-secure\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to keep your Prometheus ecosystem secure\" \/>\n<meta property=\"og:description\" content=\"Prometheus is a popular open-source monitoring system that helps track and alert for various applications. However, regarding security, there are some essential things to consider. Prometheus server, for instance, does not provide authentication or TLS encryption out of the box, meaning anyone accessing the server’s HTTP endpoints can access all of our time-series data. So, […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/how-to-keep-your-prometheus-ecosystem-secure\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2023-03-28T13:25:51+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-03-31T12:20:48+00:00\" \/>\n<meta name=\"author\" content=\"Middleware Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Middleware Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/how-to-keep-your-prometheus-ecosystem-secure\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/how-to-keep-your-prometheus-ecosystem-secure\\\/\"},\"author\":{\"name\":\"Middleware Team\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/8d8563acfc6e604cce6507f45bac0ea1\"},\"headline\":\"How to keep your Prometheus ecosystem secure\",\"datePublished\":\"2023-03-28T13:25:51+00:00\",\"dateModified\":\"2023-03-31T12:20:48+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/how-to-keep-your-prometheus-ecosystem-secure\\\/\"},\"wordCount\":754,\"commentCount\":0,\"keywords\":[\"Authentication\",\"Monitoring\",\"Prometheus\",\"secure\",\"TLS\"],\"articleSection\":[\"Database Administration & Monitoring\",\"DevOps\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/how-to-keep-your-prometheus-ecosystem-secure\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/how-to-keep-your-prometheus-ecosystem-secure\\\/\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/how-to-keep-your-prometheus-ecosystem-secure\\\/\",\"name\":\"How to keep your Prometheus ecosystem secure - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\"},\"datePublished\":\"2023-03-28T13:25:51+00:00\",\"dateModified\":\"2023-03-31T12:20:48+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/8d8563acfc6e604cce6507f45bac0ea1\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/how-to-keep-your-prometheus-ecosystem-secure\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/how-to-keep-your-prometheus-ecosystem-secure\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/how-to-keep-your-prometheus-ecosystem-secure\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to keep your Prometheus ecosystem secure\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/#\\\/schema\\\/person\\\/8d8563acfc6e604cce6507f45bac0ea1\",\"name\":\"Middleware Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g\",\"caption\":\"Middleware Team\"},\"url\":\"https:\\\/\\\/www.dbi-services.com\\\/blog\\\/author\\\/middleware-team\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How to keep your Prometheus ecosystem secure - dbi Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/how-to-keep-your-prometheus-ecosystem-secure\/","og_locale":"en_US","og_type":"article","og_title":"How to keep your Prometheus ecosystem secure","og_description":"Prometheus is a popular open-source monitoring system that helps track and alert for various applications. However, regarding security, there are some essential things to consider. Prometheus server, for instance, does not provide authentication or TLS encryption out of the box, meaning anyone accessing the server’s HTTP endpoints can access all of our time-series data. So, […]","og_url":"https:\/\/www.dbi-services.com\/blog\/how-to-keep-your-prometheus-ecosystem-secure\/","og_site_name":"dbi Blog","article_published_time":"2023-03-28T13:25:51+00:00","article_modified_time":"2023-03-31T12:20:48+00:00","author":"Middleware Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Middleware Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/how-to-keep-your-prometheus-ecosystem-secure\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/how-to-keep-your-prometheus-ecosystem-secure\/"},"author":{"name":"Middleware Team","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1"},"headline":"How to keep your Prometheus ecosystem secure","datePublished":"2023-03-28T13:25:51+00:00","dateModified":"2023-03-31T12:20:48+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/how-to-keep-your-prometheus-ecosystem-secure\/"},"wordCount":754,"commentCount":0,"keywords":["Authentication","Monitoring","Prometheus","secure","TLS"],"articleSection":["Database Administration & Monitoring","DevOps"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/how-to-keep-your-prometheus-ecosystem-secure\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/how-to-keep-your-prometheus-ecosystem-secure\/","url":"https:\/\/www.dbi-services.com\/blog\/how-to-keep-your-prometheus-ecosystem-secure\/","name":"How to keep your Prometheus ecosystem secure - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2023-03-28T13:25:51+00:00","dateModified":"2023-03-31T12:20:48+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1"},"breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/how-to-keep-your-prometheus-ecosystem-secure\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/how-to-keep-your-prometheus-ecosystem-secure\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/how-to-keep-your-prometheus-ecosystem-secure\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"How to keep your Prometheus ecosystem secure"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/8d8563acfc6e604cce6507f45bac0ea1","name":"Middleware Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ddcae7ba0f9d1a0e7ae707f0e689e4a9c95bb48ec49c8e6d9cc86d43f4121cb6?s=96&d=mm&r=g","caption":"Middleware Team"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/middleware-team\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/23653","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/40"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=23653"}],"version-history":[{"count":11,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/23653\/revisions"}],"predecessor-version":[{"id":24080,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/23653\/revisions\/24080"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=23653"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=23653"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=23653"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=23653"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}