AWS Monitoring with Zabbix

Until recently, I did not notice that Zabbix templates can also include Hosts prototypes under (Low Level) Discovery Rules. I wondered why, so I decided to look at pre-existing templates to check what is the aim for it. I chose to focus on “AWS on HTTP” template.

“AWS on HTTP” Template

The documentation is clear enough and straight forward for the AWS setup part so I will not go too much into details there.

I simply created a monitoring user named zabbix to which I associate an Access key. Do not forget to keep these in a safe place for further usage.

I also add a permission policy as per documentation:

Zabbix Host Creation

Next, in Zabbix Front End, I create a new host:

The host is considered fake as it does not have any interface declared as you can see on screenshot (no agent, no JMX, no SNMP). It is just linked to AWS by HTTP template and belongs to Cloud Host group.

Subsequently, we must define few macros as most of the templates do require:

The access key and secret access key must be set as well as the AWS region. As this AWS account is for sanboxing, it currently does not have any active host, thus I also customized {$AWS.ECS.LLD.FILTER.STATUS.MATCHES} another macro to get all hosts (default value is ACTIVE).

Discovery Rule

AWS by HTTP template has 4 discovery rules:

• EC2 instances discovery
• ECS clusters discovery
• RDS instances discovery
• S3 buckets discovery

EC2 instances discovery

When first rule ran, it will create new hosts (even not ACTIVE) in Zabbix configuration based on the host prototypes:

And, above all, it will associate a template to each discovered EC2 instance: AWS EC2 by HTTP.

It will inherit the template macros required for authentication to AWS API as well as adding one more named {$AWS.EC2.INSTANCE.ID}. This one will be used to get specific to host items, create triggers, etc.

That is really great when monitoring a always moving cloud environment where provisioning and decommissioning of virtual machines is daily operation.

S3 buckets discovery

After few minutes, I noticed that S3 buckets discovery rule was in “not supported” status which means it was not able to gather data. The “i” icone was showing:

Cannot find the "data" array in the received JSON object.

I noticed the following error in the zabbix_server.log file:

2184:20231010:131047.104 [ AWS S3 ] ERROR: Error: cannot get URL: Timeout was reached.

Why is that and which timeout expired? This is a bit difficult to find has there are many timeout settings in Zabbix. By default, the Discovery rule time is set to 15 seconds. I tried to set it to 60:

Unfortunately, it still fails as there are too many buckets to discover. 60 seconds is the current maximum. Next release of Zabbix, ie. 7.0, will push that limit as Kaspars Mednis presented during last Zabbix Summit to 10 minutes which will surely resolve this issue.

A small feedback after more testing made today as I was able to make the discovery rule complete successfully. I compared API request/response timings between today and when it was timing out. Here is a table showing the time taken (in seconds) by each action type:

It looks like AWS was somehow throttling my API requests. Whatever the case may be, the timeout limit increase will be welcome.