Zabbix IRC Notifications

Some months ago I rolled out the terrifyingly fancy monitoring platform Zabbix to monitor all Blue Systems servers conveniently. Ever since then I wanted IRC notifications but there didn’t seem to be anything compelling available, so I got quickly annoyed and moved on.

Eventually our very own Bhushan Shah poked me enough to figure out IRC notifications.

So, now we have zabbix-irc-pusher. It is an incredibly simple script connecting to IRC and sending messages to a channel. It does so without actually demonizing, which some might argue makes the script simpler. It does however mean that the script will make numerous join/quit messages appear in the relevant IRC channels, so it is advisable to enable outside messages for that channel so the bot doesn’t actually need to join the channel.

Setting the notifications up is a bit meh though, so here’s how. This is talking about Zabbix 2.x, but all of this should largely be the same for the recently released Zabbix 3.x.

First things first. Zabbix has built-in script support that is meant to be a simple notification solution where a specific notification script is simply called with 3 arguments corresponding to an e-mail’s To, Subject and Body field. These notification scripts need to be placed into a directory your zabbix-server uses for alert scripts. You can check the zabbix_server.conf’s AlertScriptsPath variable to find or change the directory in question. By default it will be something like /usr/lib/zabbix/alertscripts/ so we are going to roll with this for now. The script in question needs to be in that directory and made executable. Once the script is working and in the correct directory all the rest of the configuration happens in Zabbix itself.

In Administration→Media types create a new media type, make it type Script and write the name of the script file.

Next you need to use the script as notification strategy for a specific user. Notifications will not be issued if your script is not actually used for notifications on any user!

Go to Administration→Users pick any enabled user and go to the Media tab. Add a new media, select your IRC notification media, set an IRC channel to send notifications to and pick the notifications that should be sent. Don’t forget to actually update the user, once you add the media.

At this point we have the notification method set up, but not the content. To do that we’ll have to configure an action. In Configuration→Actions create a new action and define content.

We use the following:

Name: Report problems to IRC
Default subject: {TRIGGER.STATUS}: {TRIGGER.NAME} - http://m.neon.kde.org/zabbix/tr_events.php?triggerid={TRIGGER.ID}&eventid={EVENT.ID}
Default message:
Recovery subject: {TRIGGER.STATUS}: {TRIGGER.NAME} - http://m.neon.kde.org/zabbix/tr_events.php?triggerid={TRIGGER.ID}&eventid={EVENT.ID}
Recovery message:

You can define a bunch of conditions in which to notify.

Last but not least, you need to associate the action with the notification method we set up. In the operations tab add a new operation and associate with the user for which you set up the notification method. Don’t forget to actually hit add for the operation and also for the action to save both.

Once you are done you should have working IRC notifications. To check simply cause an event (e.g. take an agent offline) and check the event info under Monitoring→Events. Events fitting the action conditions should now have a message actions entry with information about the message delivery and the notification should have arrived on IRC. That’s it!

Naturally, all this applies to any script based notification, so whether your script forwards the information to IRC, Telegram or perhaps even an issue tracking system doesn’t really matter as far as the Zabbix side is concerned.

Unfortunately debugging script notifications is a bit of a crafty topic, so to make sure you don’t forget anything here’s a short list of things to do:

1. Make sure Zabbix-Server has an alert scripts path set up
2. Put script in alert scripts path
3. Make script exectuable (chmod +x)
4. In Zabbix add a media with type script and the relevant script’s file name
5. Add a notification method to an enabled Zabbix user
6. Add an action and associate it with the Zabbix user
7. Check that new events have a message actions entry for the new action