InfluxDB
InfluxDB is a powerfull open source time series database (TSDB) developped by InfluxData. It’s a database optimized for time-stamped or time series data. Time series data are simply measurements or events that are tracked.
It is particully interresting for metric tracking like server cpu, ram, application performances and so on.
It is similar to SQL databases but different in many ways. The key here is the time. The database engine is optimized for high ingest and data compression. It is wrtten in Go and compiles in one single binary without any dependencies.
We can use it through a cli but the most interesting part is it’s HTTP API which will allow us to perform actions on the measurements without any third party api.
It can be installed on a single server or in the cloud. It is highly compatible with Telegraf, a time series data collector, but we will discuss it on another blog.
Installation
I did the installation on a CentOS but it can be installed on other linux distributions, OS X and windows. You can download the rpm/zip/tar on the download page or you can install it directly like following:
Add the repo:
cat <<EOF | sudo tee /etc/yum.repos.d/influxdb.repo [influxdb] name = InfluxDB Repository - RHEL $releasever baseurl = https://repos.influxdata.com/rhel/$releasever/$basearch/stable enabled = 1 gpgcheck = 1 gpgkey = https://repos.influxdata.com/influxdb.key EOF
Ensure the cache is up to date:
sudo yum makecache fast
Install the database, we install curl in order to test the HTTP API:
sudo yum -y install influxdb vim curl
We enable the HTTP API:
sudo vim /etc/influxdb/influxdb.conf [http] enabled = true bind-address = ":8086" auth-enabled = true log-enabled = true write-tracing = false pprof-enabled = true pprof-auth-enabled = true debug-pprof-enabled = false ping-auth-enabled = true
We start and enable the service:
sudo systemctl start influxdb && sudo systemctl enable influxdb
We need to open the port used by InfluxDB:
sudo firewall-cmd --add-port=8086/tcp --permanent sudo firewall-cmd --reload
Now we create an admin account:
curl -XPOST "http://localhost:8086/query" --data-urlencode "q=CREATE USER username WITH PASSWORD 'password' WITH ALL PRIVILEGES"
If the query executed correctly you shouldn’t have any response.
You can now test your connection with the cli:
influx -username 'username' -password 'password'
You can list the databases with:
SHOW DATABASES
To create a database do the following:
CREATE DATABASE mymetrics
And to navigate to your created DB:
USE mymetrics
Once you’ll have some entries you will be able to list them like:
SHOW MEASUREMENTS
And then you can execute basic SQL like queries (only when you have measurements):
SELECT * FROM metric_cpu_load
Measurements via Java
Now that we have our DB setup and a user to connect, we will generate measurements with Java. But you can do it with any HTTP request compatible languages. It exists a Java api to discuss with the database directly (here) but I wanted to do it through the HTTP API.
You will need httpclient.jar and httpcore.jar from Apache Commons to use the latest http api. The version I used was:
- httpclient5-5.0.1.jar
- httpcore5-5.0.1.jar
String data = "metric_cpu_load value=0.5" // We create a default HTTP client HttpClient httpclient = HttpClients.createDefault(); // We create the URI with all informations URI uri = new URIBuilder() .setScheme("http") // Can be https if ssl is enabled .setHost("localhost") // The hostname of the database .setPort(8086) // Default InfluxDB port .setPath("/write") // We call write to put data into the database .setParameter("db", "mymetrics") // The name of the database we created .setParameter("u", "username") // The username of the account we created .setParameter("p", "password") // The password of the account .build(); HttpPost httppost = new HttpPost(uri); // We create a POST request in order to add data StringEntity entity = new StringEntity(data, ContentType.create("plain/text")); // The metric name and value is set as the body of the post request httppost.setEntity(entity); // We link the entity to the post request httpclient.execute(httppost); // Executes the post request, the result is an HttpResponse object, if the query went well, you should have a code 204
InfluxDB stores measurements in “tables”. One measure type is a table, you can specify several columns in the table, but note that the “value” column is mandatory. Here is the explanation of the body of the post request:
“table_name, col1=value1, col2=value2 value=value0 timestamp”
- table_name is the name of the metric, if the table doesn’t exist it will be created, if it exists, a new row will be add to this “table”
- colx are optionnal, they are tags or info that will specify this row, you can even have some rows with theses options while others don’t
- value is the mandatory field, it must be specified or the post request will fail
- timestamp is the timestamp of the row, you can specify it or remove it. If not specified, it will be set to the time when the row was added
Example
Here a simple example
We register several values:
[POST] http://localhost:8086/write?db=mymetrics&u=username&p=password BODY: "metric_cpu_load,hostname=localhost value=0.1" [POST] http://localhost:8086/write?db=mymetrics&u=username&p=password BODY: "metric_cpu_load,hostname=localhost value=0.4" [POST] http://localhost:8086/write?db=mymetrics&u=username&p=password BODY: "metric_cpu_load,hostname=localhost value=0.8" [POST] http://localhost:8086/write?db=mymetrics&u=username&p=password BODY: "metric_cpu_load,hostname=localhost value=0.5"
Or in CURL we create the DB if it doesn’t exist:
curl -i -XPOST 'http://localhost:8086/query?u=username&p=password' --data-urlencode "q=CREATE DATABASE mymetrics" -> returns code 200 OK curl -i -XPOST 'http://localhost:8086/write?db=mymetrics&u=username&p=password' --data-binary 'metric_cpu_load,hostname=localhost value=0.1' -> returns 204 NO CONTENT curl -i -XPOST 'http://localhost:8086/write?db=mymetrics&u=username&p=password' --data-binary 'metric_cpu_load,hostname=localhost value=0.4' curl -i -XPOST 'http://localhost:8086/write?db=mymetrics&u=username&p=password' --data-binary 'metric_cpu_load,hostname=localhost value=0.8' curl -i -XPOST 'http://localhost:8086/write?db=mymetrics&u=username&p=password' --data-binary 'metric_cpu_load,hostname=localhost value=0.5'
Now you can see the result from the cli:
influx -username 'username' -password 'password' Connected to http://localhost:8086 version 1.8.1 InfluxDB shell version: 1.8.1 > SHOW DATABASES name: databases name ---- _internal mymetrics > USE mymetrics Using database mymetrics > SHOW MEASUREMENTS name: measurements name ---- metric_cpu_load > SELECT * FROM metric_cpu_load name: metric_cpu_load time hostname value ---- -------- ----- 1595318363624096578 localhost 0.1 1595318430152497136 localhost 0.4 1595318433209434527 localhost 0.8 1595318436384650878 localhost 0.5
We can also get it from HTTP:
http://localhost:8086/query?db=mymetrics&u=username&p=password&q=select * from metric_cpu_load
{"results": [{ "statement_id":0, "series":[{ "name":"metric_cpu_load", "columns":["time","hostname","value"], "values":[ ["2020-07-21T07:59:23.624096578Z","localhost",0.1], ["2020-07-21T08:00:30.152497136Z","localhost",0.4], ["2020-07-21T08:00:33.209434527Z","localhost",0.8], ["2020-07-21T08:00:36.384650878Z","localhost",0.5]]}]}]}
Next steps
And now what? We have a time based database where we can list and add measurements through Java and CURL, what can we do with this?
Next time we will see how to display the results in shiny graphs thanks to Grafana.