How to keep your Prometheus ecosystem secure

Prometheus is a popular open-source monitoring system that helps track and alert for various applications. However, regarding security, there are some essential things to consider.
Prometheus server, for instance, does not provide authentication or TLS encryption out of the box, meaning anyone accessing the server’s HTTP endpoints can access all of our time-series data.

So, what solutions are available?
One solution would be to secure the Prometheus server accesses using a reverse proxy, encrypting the communications between clients and a server. 
We can implement this security layer with an Apache or Nginx server configured as a reverse proxy that implements TLS encryption, authentication and other security features.

When it comes to exporters, many of them do provide some security features such as authentication and/or TLS encryption, and we can set up your Prometheus server configuration to interact with those secure exporters.
This blog post will focus on securing communications between exporters and the Prometheus server by implementing authentication and TLS encryption. I will dedicate a separate blog post for securing the Prometheus server accesses using a reverse proxy like NGINX.

Basic Auth

Let’s assume we have a configuration with a Prometheus server that monitors one server with the Node Exporter configured as a service with minimal configuration. The idea is first to implement authentication and then TLS encryption.

To do this, we will create a configuration file containing our exporter’s encrypted credentials.

First, log in to our node01, and install the apache2-utils package, which we will use to create a hash for the password.

apt update
apt install apache2-utils -y

Generate the password hash

htpasswd -nBC 10 "" | tr -d ':\n'; echo

The system will prompt us to enter our password twice, as shown below:

New password:
Re-type new password:

Afterwards, a hashed value of our password will be provided. We can then proceed to edit the configuration file located at /etc/node_exporter/config.yml and add the lines below (of course, replace the placeholder by the hashed password generated in the previous step):

basic_auth_users:
  prometheus: ###our-hashed-password###

With the node_exporter configuration file created, it is necessary to declare it in the node_exporter  systemd service file so that it is considered.
For this, we have to edit the /etc/systemd/system/node_exporter.service and add the argument --web.config in the ExecStart line, as below:

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=nodexporter
Group=nodexporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --web.config=/etc/node_exporter/config.yml

[Install]
WantedBy=multi-user.target

Reload the daemon and Restart the node_exporter service:

systemctl daemon-reload
systemctl restart node_exporter

We can confirm the modifications using the “curl” command to access the metrics curl http://dbinode01:9100/metrics. The output should return Unauthorize.
We must provide the username and password in our curl command to access the metrics.

curl -u prometheus:<clear-password> http://dbinode01:9100/metrics

Now that authentication has been implemented at the exporter level, and we need to configure it at the Prometheus server level.
Without this, the job that scrapes metrics from our node_exporter will return a “401 Unauthorized” error. To do this, log in to our Prometheus server, edit the configuration file, and add the following configuration to the job that is supposed to collect metrics from our node exporter (which we will refer to as “nodes” here).

On our Prometheus server, edit the Prometheus configuration file

vi /etc/prometheus/prometheus.yml

Add the following line under - job_name: "nodes"

basic_auth:
  username: prometheus
  password: clear-password

Restart our Prometheus service:

systemctl restart prometheus

TLS encryption

With authentication in place, we will now focus on encrypting the communication between the Prometheus server and its exporter.

Let’s start by generating a certificate and its key on our node01 by running the following command:

openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout node_exporter.key -out node_exporter.crt -subj "/C=CH/ST=Switzerland/L=Basel/O=dbi/CN=localhost" -addext "subjectAltName = DNS:localhost"

Move the crt and the key file under the node_exporter folder /etc/node_exporter/

mv node_exporter.crt node_exporter.key /etc/node_exporter/

Change the ownership:

chown nodexporter.nodexporter /etc/node_exporter/node_exporter.key
chown nodexporter.nodexporter /etc/node_exporter/node_exporter.crt

Edit node_exporter configuration file /etc/node_exporter/config.yml and add the following block:

tls_server_config:
  cert_file: node_exporter.crt
  key_file: node_exporter.key

Restart the node_exporter service to load the new configuration

systemctl restart node_exporter

We can confirm our modifications by using the “curl” command with -k flag to allow connections to insecure HTTPS servers.

curl -u prometheus:<clear-password> -k https://dbinode01:9100/metrics

The finish line is in sight, and we need to configure the Prometheus server to use HTTPS for scraping the node_exporter.
To do this, we need to retrieve the certificate and its key, place them in the Prometheus directory, and update the scraping job configuration in the Prometheus configuration to allow TLS communication.

Let’s copy the certificates to our Prometheus server and change the ownership.

scp root@dbinode01:/etc/node_exporter/node_exporter.crt /etc/prometheus/node_exporter.crt
chown prometheus:prometheus /etc/prometheus/node_exporter.crt

Edit /etc/prometheus/prometheus.yml file and add the lines provided below under - job_name: "nodes"

    scheme: https
    tls_config:
      ca_file: /etc/prometheus/node_exporter.crt
      insecure_skip_verify:true

Restart Prometheus service

systemctl restart prometheus

And that’s it.

Conclusion

That’s how we can establish authentication and encryption between our exporters and the Prometheus server.

It’s crucial to take security seriously and carefully consider the security implications of our Prometheus ecosystem.