ksqlDB

ksqlDB is a server that takes benefit of an Apache Kafka infrastructure for real time data streaming. It can be used to capture events (via, for example, Kafka Connect), transform events, expose views (or tables).

Starting ksqlDB server is easy. First, we need to set the bootstrap servers of our Kafka cluster in etc/ksqldb/ksql-server.properties file:

bootstrap.servers=localhost:29092

As it is used for a proof of concept, I use a one broker cluster (ie. without high availability). Of course, ksqlDB server supports HA.

The command to start:

bin/ksql-server-start etc/ksqldb/ksql-server.properties

And once, it up and running, you should see this:

[2024-04-19 11:43:25,485] INFO Waiting until monitored service is ready for metrics collection (io.confluent.support.metrics.BaseMetricsReporter:173)
[2024-04-19 11:43:25,485] INFO Monitored service is now ready (io.confluent.support.metrics.BaseMetricsReporter:185)
[2024-04-19 11:43:25,485] INFO Attempting to collect and submit metrics (io.confluent.support.metrics.BaseMetricsReporter:144)
[2024-04-19 11:43:25,486] INFO ksqlDB API server listening on http://0.0.0.0:8088 (io.confluent.ksql.rest.server.KsqlRestApplication:385)

                  ===========================================
                  =       _              _ ____  ____       =
                  =      | | _____  __ _| |  _ \| __ )      =
                  =      | |/ / __|/ _` | | | | |  _ \      =
                  =      |   <\__ \ (_| | | |_| | |_) |     =
                  =      |_|\_\___/\__, |_|____/|____/      =
                  =                   |_|                   =
                  =        The Database purpose-built       =
                  =        for stream processing apps       =
                  ===========================================

Copyright 2017-2022 Confluent Inc.

Server 0.29.0 listening on http://0.0.0.0:8088

To access the KSQL CLI, run:
ksql http://0.0.0.0:8088

[2024-04-19 11:43:25,489] INFO Server up and running (io.confluent.ksql.rest.server.KsqlServerMain:153)
[2024-04-19 11:47:04,248] INFO Successfully submitted metrics to Confluent via secure endpoint (io.confluent.support.metrics.submitters.ConfluentSubmitter:146)

ksqlDB Client

Now, we are ready to start the ksqlDB client:

bin/ksql http://localhost:8088

if all went well, you should receive the ksql prompt and see the server status as RUNNING:

                  ===========================================
                  =       _              _ ____  ____       =
                  =      | | _____  __ _| |  _ \| __ )      =
                  =      | |/ / __|/ _` | | | | |  _ \      =
                  =      |   <\__ \ (_| | | |_| | |_) |     =
                  =      |_|\_\___/\__, |_|____/|____/      =
                  =                   |_|                   =
                  =        The Database purpose-built       =
                  =        for stream processing apps       =
                  ===========================================

Copyright 2017-2022 Confluent Inc.

CLI v0.29.0, Server v0.29.0 located at http://localhost:8088
Server Status: RUNNING

Having trouble? Type 'help' (case-insensitive) for a rundown of how things work!

ksql>

Stream

One advantage, and a drawback as well, is that a Kafka topic (refer to my previous blog post if you don’t know what this is) can store anything and each message can have its own format (text or binary). Schema Registry can enforce formatting rules, versioning and serialization/de-serialization information (I will cover that in another blog). ksql also enforces the formatting when defining a stream. For example, I create a order stream with 4 fields:

• order id
• customer id
• product id
• status

ksql> create stream order_stream (order_id int, customer_id int, product_id int, status_id int)
with (kafka_topic='order_topic',value_format='json',partitions=1);

 Message
----------------
 Stream created
----------------

To check what happened in the background, I could either use Kafka UI or even ksql:

ksql> show streams;

 Stream Name         | Kafka Topic                 | Key Format | Value Format | Windowed
------------------------------------------------------------------------------------------
 KSQL_PROCESSING_LOG | default_ksql_processing_log | KAFKA      | JSON         | false
 ORDER_STREAM        | order_topic                 | KAFKA      | JSON         | false
------------------------------------------------------------------------------------------
ksql> show topics;

 Kafka Topic                 | Partitions | Partition Replicas
---------------------------------------------------------------
 default_ksql_processing_log | 1          | 1
 order_topic                 | 1          | 1
---------------------------------------------------------------
ksql>

We can see our stream and the associated backend topic.

Let’s insert data in the stream:

insert into order_stream (order_id, customer_id, product_id, status_id) values (1, 10, 21, 0);

And check data is there:

ksql> select * from order_stream;
+-----------------------+-----------------------+-----------------------+-----------------------+
|ORDER_ID               |CUSTOMER_ID            |PRODUCT_ID             |STATUS_ID              |
+-----------------------+-----------------------+-----------------------+-----------------------+
|1                      |10                     |21                     |0                      |
Query Completed
Query terminated

And in the topic, what is actually stored? We can run a kafka-console-consumer.sh to see it. By the way, command must be started before inserting data or with --from-beginning option:

$ ./kafka-console-consumer.sh --bootstrap-server localhost:29092 --topic order_topic
{"ORDER_ID":1,"CUSTOMER_ID":10,"PRODUCT_ID":21,"STATUS_ID":0}

Every update of the order status will imply a new event in the order_stream.

Table

Let’s say we want to see the status name instead of the status id which has no meaning to us. It is possible to create a table which will contain both status_id and the associated status_name:

create table status (status_id int primary key, status_name varchar)
with (kafka_topic='status',value_format='json',partitions=1);

As you can see, when defining a table, we have to define a primary key. One of the main difference is when querying a table, only the last value of each primary key will be provided.

Let’s insert some data into status table:

insert into status (status_id, status_name) values (0,'Pending');
insert into status (status_id, status_name) values (1,'Processing');
insert into status (status_id, status_name) values (2,'Shipped');
insert into status (status_id, status_name) values (3,'Delivered');
insert into status (status_id, status_name) values (4,'Canceled');

And we can query it:

ksql> select * from status emit changes;
+-------------------------------------------------+-------------------------------------------------+
|STATUS_ID                                        |STATUS_NAME                                      |
+-------------------------------------------------+-------------------------------------------------+
|0                                                |Pending                                          |
|1                                                |Processing                                       |
|2                                                |Shipped                                          |
|3                                                |Delivered                                        |
|4                                                |Canceled                                         |

Oops, I see a typo in Canceled. How to correct it? By inserting a new record with the update:

insert into status (status_id, status_name) values (4,'Cancelled');

If I keep the select open, I will see the update and if I query it again, I see the fixed status_name:

ksql> select * from status emit changes;
+-------------------------------------------------+-------------------------------------------------+
|STATUS_ID                                        |STATUS_NAME                                      |
+-------------------------------------------------+-------------------------------------------------+
|0                                                |Pending                                          |
|1                                                |Processing                                       |
|2                                                |Shipped                                          |
|3                                                |Delivered                                        |
|4                                                |Cancelled                                        |

Joining Table and Stream

One interest of this is that you can join table and stream like in any SQL database to improve the result. The SQL query to create that stream is:

create stream order_stream_with_status as
  select order_id, customer_id, order_stream.status_id, product_id, status.status_name as status_name
  from order_stream left join status on order_stream.status_id = status.status_id
emit changes;

Here I create a new stream (ie. order_stream_with_status) based on an stream order_stream and joined to table status_name. “emit changes” is to see all changes (messages) from the topic.

Let’s see what is happening while selecting from this new stream when inserting in order_stream:

 select * from ORDER_STREAM_WITH_STATUS emit changes;
+------------------+------------------+------------------+------------------+------------------+
|ORDER_STREAM_STATU|ORDER_ID          |CUSTOMER_ID       |PRODUCT_ID        |STATUS_NAME       |
|S_ID              |                  |                  |                  |                  |
+------------------+------------------+------------------+------------------+------------------+
|0                 |1                 |10                |21                |Pending           |

Great! Now, we see a status name. Of course, this can be done with other columns as well. We can even use Kafka Connect to get data from a database like MySQL or Postgres.

Filtered Stream

Now, let’s say our shop bills customer when order has been shipped. Obviously, they don’t want to be notified on all events received in order_stream, thus we can create a new stream which will filter on status_id=2. The sql query for that can be:

create stream order_stream_billing as
 select order_id, customer_id, order_stream.status_id, product_id, status.status_name as status_name
 from order_stream left join status on order_stream.status_id = status.status_id
 where order_stream.status_id=2
emit changes;

We can insert few orders into order_stream:

insert into order_stream (order_id, customer_id, product_id, status_id) values (1, 10, 21, 0);
insert into order_stream (order_id, customer_id, product_id, status_id) values (2, 10, 21, 0);
insert into order_stream (order_id, customer_id, product_id, status_id) values (3, 10, 21, 0);
insert into order_stream (order_id, customer_id, product_id, status_id) values (4, 10, 21, 0);
insert into order_stream (order_id, customer_id, product_id, status_id) values (5, 10, 21, 0);

And then update their status:

insert into order_stream (order_id, customer_id, product_id, status_id) values (1, 10, 21, 1);
insert into order_stream (order_id, customer_id, product_id, status_id) values (2, 10, 21, 1);
insert into order_stream (order_id, customer_id, product_id, status_id) values (3, 10, 21, 2);
insert into order_stream (order_id, customer_id, product_id, status_id) values (4, 10, 21, 1);
insert into order_stream (order_id, customer_id, product_id, status_id) values (5, 10, 21, 2);

What will the select show? You guessed right, only two of them will be in the queue:

select * from ORDER_STREAM_billing emit changes;
+------------------+------------------+------------------+------------------+------------------+
|ORDER_STREAM_STATU|ORDER_ID          |CUSTOMER_ID       |PRODUCT_ID        |STATUS_NAME       |
|S_ID              |                  |                  |                  |                  |
+------------------+------------------+------------------+------------------+------------------+
|2                 |3                 |10                |21                |Shipped           |
|2                 |5                 |10                |21                |Shipped           |

And in the Infrastructure?

While doing all these tests, I forgot to check what was happening on the pure Kafka side. Let’s see:

ksql> show topics;

 Kafka Topic                 | Partitions | Partition Replicas
---------------------------------------------------------------
 ORDER_STREAM_BILLING        | 1          | 1
 ORDER_STREAM_WITH_STATUS    | 1          | 1
 default_ksql_processing_log | 1          | 1
 order_topic                 | 1          | 1
 status                      | 1          | 1
---------------------------------------------------------------

The two streams with upper case were created like that because topic name was not specified during the creation and, as per documentation, the upper case of table name is used as topic name.

In short, ksqlDB is part of an Extract, Transform and Load (ETL) process.

L’article Apache Kafka and ksqlDB est apparu en premier sur dbi Blog.

To have sentiment of the a review, I would rather have an opinion on that review like:

• Negative
• Positive
• Neutral

For that matter, I found the cardiffnlp/twitter-roberta-base-sentiment-latest to suite my needs for my tests.

Import Model

Elasticsearch provides the tool to import models from Hugging face into Elasticsearch itself: eland.

It is possible to install it or even use the pre-built docker image:

docker run -it --rm --network host docker.elastic.co/eland/eland

Let’s import the model:

eland_import_hub_model -u elastic -p 'password!' --hub-model-id cardiffnlp/twitter-roberta-base-sentiment-latest --task-type classification --url https://127.0.0.1:9200

After a minute, import completes:

2024-04-16 08:12:46,825 INFO : Model successfully imported with id 'cardiffnlp__twitter-roberta-base-sentiment-latest'

I can also check that it was imported successfully with the following API call:

GET _ml/trained_models/cardiffnlp__twitter-roberta-base-sentiment-latest

And result (extract):

{
  "count": 1,
  "trained_model_configs": [
    {
      "model_id": "cardiffnlp__twitter-roberta-base-sentiment-latest",
      "model_type": "pytorch",
      "created_by": "api_user",
      "version": "12.0.0",
      "create_time": 1713255117150,
...
      "description": "Model cardiffnlp/twitter-roberta-base-sentiment-latest for task type 'text_classification'",
      "tags": [],
...
          },
          "classification_labels": [
            "negative",
            "neutral",
            "positive"
          ],
...
  ]
}

Next, model must be started:

POST _ml/trained_models/cardiffnlp__twitter-roberta-base-sentiment-latest/deployment/_start

This is subject to licensing. You might face this error “current license is non-compliant for [ml]“. For my tests, I used a trial.

Filebeat Configuration

I will use Filebeat to read review.csv file and ingest it into Elasticsearch. filebeat.yml looks like this:

filebeat.inputs:
- type: log
  paths:
    - 'C:\csv_inject\*.csv'

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  protocol: "https"
  username: "elastic"
  password: "password!"
  ssl:
    ca_trusted_fingerprint: fakefp4076a4cf5c1111ac586bafa385exxxxfde0dfe3cd7771ed
  
  indices:
    - index: "csv"
  pipeline: csv

So each time a new file gets into csv_inject folder, Filebeat will parse it and send it to my Elasticsearch setup within csv index.

Pipeline

Ingest pipeline can perform basic transformation to incoming data before being indexed.

Data transformation

First step consists of converting message field, which contains one line of data, into several target fields (ie. split csv). Next, remove message field. This looks like this in Processors section of the Ingest pipeline:

Next, I also want to replace the content of the default timestamp field (ie. @timestamp) with the timestamp of the review (and remove the date field after that):

Inference

Now, I add the Inference step:

The only customization of that step is the field map as the default input field name is “text_field“, In the reviews, fields is named “comment“:

Optionally, but recommended, it is possible to add Failure processors which will set a field to keep track of the cause and will put them in a different index:

Ingest

Now, I can simply copy the review.csv into the watched directory and Filebeat will send lines to Elasticsearch. After few minutes, I can see the first results:

Or, a considered negative example with the associated prediction rate:

What Next?

Of course, we could try another model to compare results.

If you did not noticed, this was also a first step into Extract-transform-load topic (ETL).

L’article Elasticsearch, Ingest Pipeline and Machine Learning est apparu en premier sur dbi Blog.

In this blog post, I will quick explain how to setup Elasticsearch cluster, then how easy the Zabbix setup is and list possible issues you might encounter.

Elastic Cluster Setup

I will not go too much in detail as David covered already many topics around ELK. Anyway, would you need any help to install, tune or monitor your ELK cluster fell free to contact us.

My 3 virtual machines are provisioned with YaK on OCI. Then, I install the rpm on all 3 nodes.

After starting first node service, I am generating an enrollment token with this command:

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -node

This return a long string which I will need to pass on node 2 and 3 of the cluster (before starting anything):

/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <...>

Output will look like that:

This node will be reconfigured to join an existing cluster, using the enrollment token that you provided.
This operation will overwrite the existing configuration. Specifically:
  - Security auto configuration will be removed from elasticsearch.yml
  - The [certs] config directory will be removed
  - Security auto configuration related secure settings will be removed from the elasticsearch.keystore
Do you want to continue with the reconfiguration process [y/N]

After confirming with a y, we are almost ready to start. First, we must update ES configuration file (ie. /etc/elasticsearch/elasticsearch.yml).

• Add IP of first node (only for first boot strapped) in cluster.initial_master_nodes: ["10.0.0.x"]
• Set listening IP of the inter-node trafic (to do on node 1 as well): transport.host: 0.0.0.0
• Set list of master eligible nodes: discovery.seed_hosts: ["10.0.0.x:9300"]

Now, we are ready to start node 2 and 3.

Let’s check the health of our cluster:

curl -s https://localhost:9200/_cluster/health -k -u elastic:password | jq

If you forgot elastic password, you can reset it with this command:

/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

Zabbix Configuration

With latest Elasticsearch release, security has drastically increased as SSL communication became the standard. Nevertheless, default MACROS values of the template did not. Thus, we have to customize the followings:

• {$ELASTICSEARCH.USERNAME} to elastic
• {$ELASTICSEARCH.PASSWORD} to its password
• {$ELASTICSEARCH.SCHEME} to https

If SELinux is enabled on your Zabbix server, you will need to allow zabbix_server process to send network request. Following command achieves this:

setsebool zabbix_can_network 1

Next, we can create a host in Zabbix UI like that:

The Agent interface is required but will not be used to reach any agent as there are not agent based (passive or active) checks in the linked template. However, http checks uses HOST.CONN MACRO in the URLs. Ideally, the IP should be a virtual IP or a load balanced IP.

Don’t forget to set the MACROS:

After few minutes, and once nodes discovery ran, you should see something like that:

Problems

What will happen if one node stops? On Problems tab of Zabbix UI:

After few seconds, I noticed that ES: Health is YELLOW gets resolved on its own. Why? Because shards are re-balanced across running servers.

I confirm this by graphing Number of unassigned shards:

We can also see the re-balancing with the network traffic monitoring:

Around 15:24, I stopped node 3 and shards were redistributed from node 1 and 2.

When node 3 start, at 15:56, we can see node 1 and 2 (20 Mbps each) send back shards to node 3 (40 Mbps received).

Conclusion

Whatever the monitoring tool you are using, it always help to understand what is happening behind the scene.

L’article Monitor Elasticsearch Cluster with Zabbix est apparu en premier sur dbi Blog.

Setup Zabbix Monitoring

Apache httpd template provided by Zabbix uses mod_status which provides metrics about load, processes and connections.

Before enabling this module, we must ensure it is present. httpd -M 2>/dev/null | grep status_module command will tell you so. Next, we can extend configuration by creating a file in /etc/httpd/conf.d:

<Location "/server-status">
  SetHandler server-status
</Location>

After a configuration reload, we should be able to access the URL http://<IP>/server-status?auto.

Finally, we can link the template to the host and see that data are collected:

Tuning

I deployed a simple static web site to the Apache httpd server. To load test that web site, nothing better than JMeter. The load test scenario is simply requesting Home, About, Services and Contact Us pages and retrieve all embedded resources during 2 minutes with 100 threads (ie. users).

Here are the performances on requests per seconds (right scale) and bytes per seconds (left scale):

At most, server serves 560 req/s at 35 MBps.

And regarding CPU usage, it almost reaches 10%:

Compression

Without any additional headers, Apache httpd will consider the client (here JMeter) does not support gzip. Fortunately, it is possible to set HTTP Header in JMeter. I add it at the top of the test plan so that it will apply to all HTTP Requests below:

Note that I enabled mod_deflate on Apache side.

Let’s run another load test and compare the results!

After two minutes, here is what I see:

The amount of Mbps reduced to 32 which is expected as we are compressing. The amount of req/s increased by almost 100% to 1000 req/s !

On the CPU side, we also see a huge increase:

This is also more or else expected as compression requires computing.

And Now

The deployed static web site does not have any forms which would require client side compression. That will be a subject for another blog. Also, I can compare with Nginx.

L’article Apache httpd Tuning and Monitoring with Zabbix est apparu en premier sur dbi Blog.

In this blog post, I will cover one of them and run different scenarios. We will also dig inside some internal mechanism used by Ansible.

SSH Connections

Ansible is connection intensive as it opens, and closes, many ssh connections to the targeted hosts.

I found two possible ways to count the amount of connections from control to agents nodes:

• Add -vvv option to the ansible-playbook command.
• grep audit.log file:

tail -f /var/log/audit/audit | grep USER_LOGIN

First option is really too much verbose, but I used it with the first playbook below to confirm the second option give the same count.

Simple Playbook

To demonstrate that, let’s start with a very minimal playbook without fact gathering:

---
- name: Test playbook
  hosts: all
  gather_facts: false
  pre_tasks:
    - name: "ping"
      ansible.builtin.ping:
...

This playbook triggered 8 ssh connections to the target host. If I enable facts gathering, count goes to 14 connections. Again, this is quiet a lot knowing that playbook does not do much beside check target is alive.

To summarize:

What are All These Connection For?

To determine what are these connections doing, we can analyze verbose (really verbose!!) output of Ansible playbook without fact gathering.

First Connection

First command of first connection is echo ~opc && sleep 0 which will return the home directory of ansible user.

Second

Second command is already scary:

( umask 77 && mkdir -p "` echo /home/opc/.ansible/tmp `"&& mkdir "` echo /home/opc/.ansible/tmp/ansible-tmp-1712570792.3350322-23674-205520350912482 `" && echo ansible-tmp-1712570792.3350322-23674-205520350912482="` echo /home/opc/.ansible/tmp/ansible-tmp-1712570792.3350322-23674-205520350912482 `" ) && sleep 0

1. It set the umask for the commands to follow.
2. Create tmp directory to store any python script on target
3. In that directory, create a directory to store the script for this specific task
4. Makes this ssh command return the temporary variable with full path to the task script directory
5. sleep 0

This one is mainly to ensure directory structure exists on the target.

Third

I will not paste this one here as it is very long and we can easily guess what it does with log just before:

Attempting python interpreter discovery

Roughly, what it does, it tries many versions of python.

Fourth

Next, it will run a python script with discovered python version to determine Operating System type and release.

Fifth

Fifth connection is actually a sftp command to copy module content (AnsiballZ_ping.py). AnsiballZ is a framework to embed module into script itself. This allows to be run modules with a single Python copy.

Seventh

This one is simply ensuring execution permission is set on temporary directory (ie. ansible-tmp-1712570792.3350322-23674-205520350912482) as well the python script (ie. AnsiballZ_ping.py).

Eighth and Last Connection

Lastly, the execution of the ping module itself:

/usr/bin/python3.9 /home/opc/.ansible/tmp/ansible-tmp-1712570792.3350322-23674-205520350912482/AnsiballZ_ping.py && sleep 0

Optimization

To reduce the amount of connection, there is one possible option: Pipelining

To enable that, I simply need to add following line in ansible.cfg:

pipelining = true

Or set ANSIBLE_PIPELINING environment variable to true.

How does it improve our playbook execution time:

As we can see there is a significant reduction on the amount of ssh connections as well as a reduction of the playbook duration.

In this configuration, only 3 connections are made:

• python interpreter discovery (connection #3)
• OS type discovery (connection #4)
• python module execution (connection #8). AnsiballZ data is piped to that process.

With the pipelining option, I also noticed that the Ansible temporary directory is not created.

Of course, we can’t expect such big speed-up on a real life playbook. So, we should do it now.

Deploy WebLogic Server Playbook

Let’s use the WebLogic YaK component to deploy a single WebLogic instance. It includes dbi service best practices, latest CPU patches and SSL configuration. The “normal” run takes 13 minutes 30 seconds when the pipelined run takes 12 minutes 13 seconds. This is 10% faster.

This is nice, but not as good as previous playbook. Why is that? Because most of the time is not spent in ssh connections, but with actual work (running WebLogic installer, starting services, patching with OPatch, etc).

What Next?

With such results, you might wonder why isn’t it enabled by default? As per documentation, there is a limitation:

This can conflict with privilege escalation (become). For example, when using sudo operations you must first disable ‘requiretty’ in the sudoers file for the target hosts, which is why this feature is disabled by default.

Ansible documentation

Until now, with all tests I have made, I never encountered that limitation. Did you?

L’article Faster Ansible est apparu en premier sur dbi Blog.

In this blog post, we will see how easy it is to monitor a Kubernetes cluster with Zabbix.

Monitoring Architecture

As Kubernetes is a container orchestration system for automating software deployment […] (wikipedia), you don’t want to manually install a Zabbix agent on each node of the cluster. With that in mind, Zabbix developed a helm chart to deploy what is required to monitor the cluster:

• A DaemonSet on all cluster nodes for Zabbix agent (one agent per node)
• A single replica deployment for the Zabbix proxy who will act as a collection point for all Zabbix agents

The Zabbix proxy will run in ActiveMode meaning that it will get configuration from Zabbix server and push data to it.

Kubernetes Preparation

Among all the customizable values of the helm chart, the most important is the Zabbix server IP/hostname as Zabbix proxy requires it to push data to Zabbix server:

    - name: ZBX_SERVER_HOST
      value: "192.168.x.y"

Once the chart has been installed in the dedicated monitoring namespace of your Kubernetes cluster, we should be able to see the pods:

$ kubectl get pods -n monitoring
NAME                                         READY   STATUS    RESTARTS   AGE
zabbix-agent-lbwsg                           1/1     Running   0          15m
zabbix-kube-state-metrics-64aac65c45-g9n94   1/1     Running   0          15m
zabbix-proxy-5b4faaff64-xf8wk                1/1     Running   0          15m

Zabbix Web UI

On Zabbix side, we must create the zabbix proxy:

Next, we can create a “fake” host as it does not really correspond to any host. It is just a place holder for the configuration that must be provided to Zabbix proxy:

There is no need to set an interface as the selected template will not use it anyway. In the Macros tab, we must set two of them:

• {$KUBE.API.URL}
• {$KUBE.API.TOKEN}

Thus, it will look like this:

After a couple of minutes, you will see a new host with Zabbix agent in green:

The IP 192.168.49.2 is an IP internal the Kubernetes cluster (here provisioned with minikube on a raspberry pi 4).

L’article Monitor Kubernetes with Zabbix est apparu en premier sur dbi Blog.

Setup community.zabbix

The first step is easy and fast. In your YaK environment, run the following command:

ansible-galaxy collection install community.zabbix

This collection is actually an interface with Zabbix API which make our work easier when we already know Ansible and don’t want to go through complex HTTP API calls (see my other blog post about it).

Setup Ansible Variables

If you know Ansible, you know that all tasks are going through a ssh connection to the target host. Once agent is installed on target host, we want to configure it in Zabbix server. This is achieved by modifying few variables before calling zabbix modules:

- name: Set api configuration facts
  ansible.builtin.set_fact:
    ansible_network_os: community.zabbix.zabbix
    ansible_connection: httpapi
    ansible_httpapi_port: 80
    ansible_zabbix_url_path: ''
    ansible_host: "{{ zabbix_server_ip }}" # Zabbix server
    ansible_user: "{{ zabbix_api_login_user }}"

For requests to be redirected, all tasks concerning our host must go to the zabbix server (line 7). Zabbix server does not understand ssh connection, but httpapi connections (line 4) with ansible_network_os specific to the API (line 3). Authentication against the http api is made with ansible_user as login and ansible_httpapi_pass as password. For better security, we could create an API token and use https.

Usage

Host Group

Before creating our host, we can start with a simpler task like creating a host group:

- name: Create host groups
  community.zabbix.zabbix_group:
    host_group: "{{ zabbix_agent_groups }}"
  register: zabbix_api_hostgroup_created
  until: zabbix_api_hostgroup_created is succeeded

This task calls zabbix_group module with a list of group in host_group argument. Could it be more easier than that?

Note that lines 4 and 5 are there to ensure host group is created as concurrent access (while running against multiple hosts) might raise creation failure. The default value for retries is 3 and with a delay of 5 seconds between each retry.

Host

To create a host in Zabbix configuration, there are a bit more variables to provide to the module:

- name: Create host in zabbix server configuration
  community.zabbix.zabbix_host:
    host_name: "{{ hostname }}"
    host_groups: "{{ zabbix_agent_groups }}"
    link_templates: "{{ zabbix_agent_link_templates }}"
    interfaces: "{{ zabbix_agent_interfaces }}"
    tags: "{{ zabbix_agent_tags }}"
    tls_connect: 2 # PSK
    tls_psk_identity: "{{ zabbix_agent2_tlspskidentity }}"
    tls_psk: "{{ zabbix_agent2_tlspsk_secret }}"
  register: zabbix_api_host_created
  until: zabbix_api_host_created is succeeded

host_name is the how the host will be named in Zabbix configuration (coming from inventory – line 3). host_groups is the list of groups we’ve created previously. link_templates is a list of Zabbix templates.

interfaces (line 6) will be the list of defined interfaces to reach the Zabbix agent from the Zabbix server. This variable content looks like:

    zabbix_agent_interfaces:
      - type: 1
        main: 1
        useip: "{{ zabbix_useuip }}"
        ip: "{{ ansible_host }}"
        dns: ""
        port: "{{ zabbix_agent_listenport }}"

Next tls_* variables are set to encrypt trafic between agent and server.

tags variable allows to have host added with tags in Zabbix configuration.

Et voilà!

L’article Configure Zabbix with Ansible est apparu en premier sur dbi Blog.

In today’s business landscape, everyone is facing a crucial dilemma: how to store my information? How to secure my data? How to manage my content processes? How to make my data more intelligent? Which solution to use?

In this blog, I will show you that, whatever the size of the business, a company should use an ECM/IIM solution to digitally manage the information to support its processes and business goals.

What is it?

An ECM – Enterprise Content Management – is a combination of methods and strategies used to collect, manage, and deliver information in various forms. Information is also stored and archived for future use.

But ECM is not able to manage and analyze unstructured content like emails, rich media, invoices and other records, analytics and more. That’s where IIM arrived as a rebirth of ECM.

IIM – Intelligent Information Management – is the solution to address the growing demands for increased capacity, seamless information integration, time efficiency, and cost savings. In order for you to access all the data you were missing before.

What does it do?

An ECM/IIM solution is not only scanning, storing and retrieving information but a strategic system for collecting data and organizing information. It digitalizes, controls and automates the flow of unstructured information within an organization. It also captures, reads, analyzes and indexes the information. You can annotate, edit or improve the information. It offers workflow and automation tools to ensure that information goes through to the correct approval processes within your organization and is assigned to the right people at the right time.

Why is it important to have such solution?

Having a lot of information to manage, it is not a simple task for a business: store the information, classify the information, retrieve the information… An ECM/IIM solution can help you do just that, and facilitate the routines for processing and analyzing your information in your company. For example, when your information is unorganized, such a solution can help you to structure it. When you need to establish different level of access on the information based on some criteria, such as the departments within your business. When you have a lot of paper format information and you want to digitalize it all.

Regardless of your business’s size or industry, an ECM/IIM solution brings you a lot of advantages!

Who is it for?

At the beginning, ECM/IIM solutions were for large companies because of the cost (infrastructure, licenses, maintenance) and the need of IT teams to implement and manage them. This is not the case anymore! Many ECM/IIM solutions are now affordable and accessible for small and medium-sized businesses, thanks to technological advances, which have virtually eliminated the need of heavy IT resources. The systems became more user-friendly and easier to implement and manage by non-IT people.

Recently, the low-code/no-code for ECM/IIM solutions are a simple and cost-effective way of developing applications without specialized people. Using a straightforward visual interface and drag-and-drop tools, you can reduce the workload on your IT teams and achieve a faster time-to-market of your ECM/IIM solution.

ECM/IIM solutions are now equally suited to all industries or businesses.

Here are some examples:

Healthcare industries has embraced ECM/IIM solutions to reduce costs and improve patient care by managing all their information.  Patient information, medications, electronic medical records (EMR), physician notes and lab results are some of the examples. Loosing time to search for patient information — or worse, not knowing it exists — could generate errors and have devastating consequences. In addition, to automate all quality and document management related processes, healthcare industries are using QMS – Quality Management System – solutions.

Banks and financial services are good candidates for ECM/IIM solutions. They manage a high-volume transactions which involves manual document processing. Such solutions digitizes the reports and documents to streamline business processes, improves the speed of business operations, decisions and enhances customer service with a complete customer view.

Administrative departments (HR, Financial, Marketing…) of any companies also needs ECM/IIM solutions to manage all the personal information of their employees (like CVs, motivation letter, contracts, certification, payslips…) but also the client or providers information (like contact info, invoices, contracts…), templates documents, internal documentation.

Retail establishment can use ECM/IIM solutions to run merchandising and supply chain functions as cost-effectively and efficiently as possible. They use it to manage their supplier contracts, orders and shipments and to better understand customers and the markets trends. Providing current and accurate product information allows to improve employee productivity. The use of ECM/IIM solutions enable them to react more quickly to changing market demands, and considerably reduce document storage and handling costs.

Manufacturing industries are also using ECM/IIM solutions to provide a unified platform for all content and data across the enterprise. It is perfect for organizing product data, processing invoices, and managing employee information. It helps managers automate complex business process to
reduce processing times within the enterprise, including project management, financial and procurement services, human resources, compliance and many other areas.

Colleges, universities and other educational institutions accumulate a lot of records which need to be secured, organized, and easily retrieved. That’s why they use ECM/IIM solutions to digitize their documents and formally manage their electronic content. They can store student and employee records digitally, handle applications with electronic workflows, preserve media and documents digitally and automate routine processes.

Governments and municipalities are also using high volumes of documents, including a lot of paper documents. Their document management needs can be unique, and errors can cause serious consequences. They have complicated processes and requirements for document flow. Using ECM/IIM solutions can help them to reduce the footprint, automate their paper workflows, ensure their document security and regulatory compliance and reduce a lot administrative tasks and unnecessary bureaucracy.

As you can see, a lot of businesses are using ECM/IIM solutions in order to control and manage their critical information. They realized that they can save time and money. Visibility and control are the key words.

Conclusion

ECM/IIM solutions can be used by any size of industries or businesses. It is a fact! Any kind of business looking for a solution to cut costs, improve productivity, and streamline business processes will benefit from ECM/IIM solutions. The amount of data generated and manipulated by businesses is increasing a lot, every year, so using such solutions is an excellent way to gain control of the content and save money in your organization, regardless of its size.

Our offer at dbi services

dbi services is specialized in the integration of content management solutions. Our expertise in database, middleware technologies and operating systems guarantees quality implementation of ECM/IIM solutions in your IT infrastructure.

We offer:

• Enhanced expertise: With a deep understanding of content management solutions, our team ensures superior quality ECM/IIM implementations.
• Increased security and compliance: We adhere to the highest standards to protect your data.
• Tailored and effective solution: We finely tune ECM/IIM solutions to your specific needs, boosting the efficiency of your organization.

Our solutions:

For more information: https://www.dbi-services.com/expertises/enterprise-content-management-ecm

L’article Why businesses – regardless of their size – have an interest in using an ECM/IIM solution? est apparu en premier sur dbi Blog.

Login

In my “M-Files by Zabbix agent active” template, I create a new scenario named “Login/Logout” where I define two variables:

On the left side, it is the name of the variable which be used in any parameters of the web request (Headers, Steps). On the right, it is a template MACRO which can then be set at host level.

Next, I go in Steps tab and add the Login step:

You might wonder “Where did he get the URL and the raw post data?” I took it from my previous recordings made with JMeter (see this blog post).

Inside that raw data, which is in JSON format, I map the variables we declared in scenario (ie. {username} and {password}):

...
            "credentials": {
                "username": "${username}",
                "password": "${password}",
                "domain": "",
                "type": "CREDENTIALS_TYPE_AUTOMATIC"
            }
...

Note that the above code is only a partial extract of Raw post content.

To determine if request is successful or not, it used the two bottom fields:

• Required string
• Required status codes

I know from experience that we expect the standard HTTP status 200 when all goes well. Nothing prevents you to add a required string like, for example, “session_data“. This string is supposed to be in the response when all goes well:

{
	"authentication_data": {
		"type": "AUTH_DATA_TYPE_CREDENTIALS",
		"data": {
			"credentials": {}
		},
		"authentication_attempt_identifier": {
			"legacy_data": 0,
			"data": "AAAAnxKxXNAAAAAAAAAAAAdID0sK/1blLGtwrfs="
		}
	},
	"session_data": {
		"session_id": "AAAAH7UQxz5P7BAAAAAAAAAmYTO+ctY+64otfz",
		"account_name": "ols1",
		"access_rights": {
...

Logout

The next step will be a Logout. The tricky part of this step is that it requires a specific header for the application to correlate the logout with the right user. This session id is in the body result of the login step. To extract it, we need to use a special tag in the value part (ie. regex:). This will look like:

After the Login step, {SID} variable will be populated with what the regex returns and can be reused in Headers section of Logout step:

Note that I also set {} as raw data and 200 as required status code.

When the two steps are defined, the Steps tab of the scenario shows an overview:

Resulting Data

Once the scenario is setup, it automatically creates a few items, along with a set of nice graphs. Items will be about:

• Download speed (per step)
• Number of failed steps
• Last error message
• Response code (per step)
• Response time (per step)

Based on these, we can create triggers to raise an alarm. For example, based on “Failed step of scenario “Login/Logout”.” item, we can imagine raising an alert as soon as there is a failed step with this expression:

last(/M-Files by Zabbix agent active/web.test.fail[Login/Logout])>0

Additionally, graphics on download speed and response time with steps distinctions:

L’article Zabbix Web Monitoring est apparu en premier sur dbi Blog.

• Extending a scenario with extra actions
• Update a scenario as a new version of the application is behaving differently (different HTTP POST data and HTTP results)

In this blog post, we will improve the existing load test scenario for M-Files.

Extend a Scenario

When extending a scenario, the HTTP requests we will add will have different session ID (sid, M-Files-Session) and thus, we must find them and do a Search and Replace again. We must repeat this every time we add an HTTP request from a different recording. This might cause problems and waste of debugging time if not done properly.

A better approach would be to add a new type of object: A JSR223 Preprocessor. This code will be called before (as “pre” indicates) each call of any type of object of the scenario. With this, we will be able to modify any HTTP request before it triggers.

If you recall my previous blog post, I discovered a client-side generated timestamp hash based on yeast library:

This is used sometimes in URL as GET parameters, sometimes in POST. We will first focus on GET.

HTTP GET

I will use a PreProcessor of type Apache Groovy which is known to be more efficient than other languages supported by JMeter.

To avoid having the code run for all types of requests, my code will start with an if:

if (sampler instanceof org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy) {

Next part of the code is to check if URL contains the “t” parameter with a regex and replace:

    path = sampler.getPath();
	findRegex = /&t=(.+)&/;
	path = path.replaceFirst(findRegex, "&t=" + newValue + "&");
    log.info("t variable updated to " + newValue);
	sampler.setPath(path);

Now, whenever I run the scenario, we will see a line in the logviewer:

2024-01-29 09:45:55,722 INFO o.a.j.m.J.JSR223 PreProcessor Variable Updates: t variable updated to OrKPOIA.

And that the triggered URL is different from how it was defined in the HTTP request:

On the left side, we see the “t” value at recording time. On the right, “t” value at scenario runtime.

This covers the case where “t” variable is in the URL. Next, we must do it for POST variables.

HTTP POST

For HTTP POST, we must proceed in the following order:

1. Check HTTP Method is POST
2. Get arguments list
3. Check our argument is in the list
4. Remove it
5. Add it back with the correct value

This gives the following groovy code:

	if (sampler.getMethod() == 'POST') {
		arguments = sampler.getArguments().getArgumentsAsMap()
		if (arguments.containsKey("t")) {
			sampler.getArguments().removeArgument("t")
			sampler.addArgument("t",newValue)
			log.info("POST: t variable updated to " + newValue);
		}
	}

What to Improve even More?

I need to make this more generic via a method which I will be able to call for any variable as I have many other variables to consider for this M-Files scenario:

• SID
• M-Files-Session
• Vault ID

Finally, for each of these, I will simply call the method:

updateVariables("t", newValue)
updateVariables("SID", vars.get("SID"))
updateVariables("M-Files-Session", vars.get("SID-MFILES"))

L’article How to Improve your JMeter Load Testing? est apparu en premier sur dbi Blog.