Adapt Zabbix to Your Needs

Zabbix comes with an long list of smart and customizable templates (more than 270). In this blog post, I will explain how to customize these templates and adapt them to your environment’s needs.

Macro

The first level of customization is the macro. There are several types of macros:

• {MACRO} built-in macro
• {<macro>.<func>(<params>)} macro functions
• {$MACRO} user-defined macro
• {#MACRO} macro for low-level discovery
• {?EXPRESSION} expression macro

For templates, it declares user-defined macros (ie. with $ sign).

By looking at Linux by Zabbix agent template, we can see that it defines 28 macros. You can either get that list from Zabbix user interface or on Zabbix web site. We can classify them in different category:

• Threshold in value:
  For example, $VFS.FS.FREE.MIN.CRIT is “the critical threshold for utilization of the filesystem” with a default value of 5G
• Threshold in percent:
  For example, $MEMORY.UTIL.MAX is a “macro used as a threshold in the memory utilization trigger” with a default value of 90 (%)
• Discovery filter:
  For example, $VFS.FS.FSNAME.NOT_MATCHES is the macro used for discovery of the filesystems. Default value is ^(/dev|/sys|/run|/proc|.+/shm$). This avoids monitoring non-relevant filesystem.

It is nice that these can be customized, but where should we do it?

If a macro is global to your environment, the default value can be changed at template level.

It is also possible to set macro at host level. Typically, when you monitor servers with different partition sizes, you don’t want to set the same thresholds. Small aside, the template trigger expression uses multiple macros to reduce customization required at the minimal.

How does it do it? By combining thresholds by percent and value. Let’s take “Disk space is low” as an example. Trigger will be raised when:

1. Partition usage (%) is above $VFS.FS.PUSED.MAX.WARN (default = 80)
   AND
2. A combination with an OR of these two conditions:
   • (Partition size – Partition used) is below $VFS.FS.FREE.MIN.WARN (default = 10G)
   • timeleft which is a function predicting when partition will reach a threshold based on past data evolution. This is also a cool feature of Zabbix 🙂

User Macros with Context

When you set a macro’s value at host level, it will be applied to all occurrences of the macro, but in some cases, you might not want to do that. A classic example is when you want to differentiate partition usage thresholds per partition on same host. If we look at trigger from previous chapter in more details, we can see that point 1 expression definition is:

last(/Linux by Zabbix agent/vfs.fs.dependent.size[{#FSNAME},pused])>{$VFS.FS.PUSED.MAX.WARN:"{#FSNAME}"}

The context here is the low level discovery macro which equals the file system name (#FSNAME). That means instead of defining a global to host value (which you can still do if desired), you can set per partition values. For example:

• $VFS.FS.PUSED.MAX.WARN = 90
• $VFS.FS.PUSED.MAX.WARN:/home = 60
• $VFS.FS.PUSED.MAX.WARN:/var/log = 75

In plain English, this means that the applied threshold will be 60% for /home, 75% for /var/log and 90% for any other discovered partitions.

Overriding Discovery Rules

Another way to have customization is to use “Overrides” feature of the Discovery Rules. Low Level Discovery, aka LLD, is one of my favorite features of Zabbix templates as it avoids manual declaration of many Zabbix entities to declare (for instance, network interfaces of server, pluggable databases of an Oracle instance). You can achieve the same goal as user macros with context, and much more complex filtering and operations.

With current version of the “Linux by Zabbix agent” template, each discovered filesystem will create 6 items based on the following 6 items prototypes:

Imagine that you don't want to monitor any NFS share mounted on the server but the "Filesystem is read-only" item as usage is already monitored through another host (the one which is hosting the filesystem).

If template is used as is, LLD will automatically create these items for /mnt/nfs mount point:

I just want to keep the one named “/mnt/nfs: Filesystem is read-only”. I must also keep “/mnt/nfs: Filesystem is read-only” as it is the master item (ie. “Filesystem is read-only” depends on “Filesystem is read-only“). Dependent item is another great feature of Zabbix to avoid unnecessary load on agent and Zabbix poller as agent is queried only once for all items on the screenshot.

Add a Discover Rule

The first step is to find where to define the new override. So, I go in the template, then Discover rules where I can find the “Mounted filesystem discovery” rule:

Next, I go in Overrides tab and click the small Add button:

Then, I can name it and add a filter. In my case, if #FSNAME macro contains nfs, I will apply it:

Now, I must add an operation in other words, what should this override do when filter is matching?

I want to avoid discovering anything that is not one of the two items. This is done by a regular expression and Discover option set to No:

That implies that the following items will pass the condition as they do not match and will not be discovered:

• Free inodes in %
• Space utilization
• Total space
• Used space

Conclusion

Clearly, the last method is more complex, but it is also much more powerful. For instance, you can customize the item update interval, storage and trend duration as well as tags on specific entities (Items, Triggers, Graphs or Hosts prototypes).