Zabbix IRC Notifications

Zabbix_logo

Some months ago I rolled out the terrifyingly fancy monitoring platform Zabbix to monitor all Blue Systems servers conveniently. Ever since then I wanted IRC notifications but there didn’t seem to be anything compelling available, so I got quickly annoyed and moved on.

Eventually our very own Bhushan Shah poked me enough to figure out IRC notifications.

So, now we have zabbix-irc-pusher. It is an incredibly simple script connecting to IRC and sending messages to a channel. It does so without actually demonizing, which some might argue makes the script simpler. It does however mean that the script will make numerous join/quit messages appear in the relevant IRC channels, so it is advisable to enable outside messages for that channel so the bot doesn’t actually need to join the channel.

Setting the notifications up is a bit meh though, so here’s how. This is talking about Zabbix 2.x, but all of this should largely be the same for the recently released Zabbix 3.x.

First things first. Zabbix has built-in script support that is meant to be a simple notification solution where a specific notification script is simply called with 3 arguments corresponding to an e-mail’s To, Subject and Body field. These notification scripts need to be placed into a directory your zabbix-server uses for alert scripts. You can check the zabbix_server.conf’s AlertScriptsPath variable to find or change the directory in question. By default it will be something like /usr/lib/zabbix/alertscripts/ so we are going to roll with this for now. The script in question needs to be in that directory and made executable. Once the script is working and in the correct directory all the rest of the configuration happens in Zabbix itself.

In Administration→Media types create a new media type, make it type Script and write the name of the script file.

zabbix-irc-01

Next you need to use the script as notification strategy for a specific user. Notifications will not be issued if your script is not actually used for notifications on any user!

Go to Administration→Users pick any enabled user and go to the Media tab. Add a new media, select your IRC notification media, set an IRC channel to send notifications to and pick the notifications that should be sent. Don’t forget to actually update the user, once you add the media.

zabbix-irc-02

At this point we have the notification method set up, but not the content. To do that we’ll have to configure an action. In Configuration→Actions create a new action and define content.

We use the following:

Name: Report problems to IRC
Default subject: {TRIGGER.STATUS}: {TRIGGER.NAME} - http://m.neon.kde.org/zabbix/tr_events.php?triggerid={TRIGGER.ID}&eventid={EVENT.ID}
Default message:
Recovery subject: {TRIGGER.STATUS}: {TRIGGER.NAME} - http://m.neon.kde.org/zabbix/tr_events.php?triggerid={TRIGGER.ID}&eventid={EVENT.ID}
Recovery message:

You can define a bunch of conditions in which to notify.

Last but not least, you need to associate the action with the notification method we set up. In the operations tab add a new operation and associate with the user for which you set up the notification method. Don’t forget to actually hit add for the operation and also for the action to save both.

zabbix-irc-04 zabbix-irc-05

Once you are done you should have working IRC notifications. To check simply cause an event (e.g. take an agent offline) and check the event info under Monitoring→Events. Events fitting the action conditions should now have a message actions entry with information about the message delivery and the notification should have arrived on IRC. That’s it!

zabbix-irc-06 zabbix-irc-07

Naturally, all this applies to any script based notification, so whether your script forwards the information to IRC, Telegram or perhaps even an issue tracking system doesn’t really matter as far as the Zabbix side is concerned.

Unfortunately debugging script notifications is a bit of a crafty topic, so to make sure you don’t forget anything here’s a short list of things to do:

  1. Make sure Zabbix-Server has an alert scripts path set up
  2. Put script in alert scripts path
  3. Make script exectuable (chmod +x)
  4. In Zabbix add a media with type script and the relevant script’s file name
  5. Add a notification method to an enabled Zabbix user
  6. Add an action and associate it with the Zabbix user
  7. Check that new events have a message actions entry for the new action

 

Building a Jenkins Security Realm

java

Last week I spent a good while on writing a new security realm for KDE’s Jenkins setups. The result of my tireless java brewing is that the Jenkins installation of KDE neon now uses KDE’s Phabricator setup to authenticate users and manage permissions via OAuth.

We should hopefully see this roll out to the KDE CI Jenkins as well in the near future.

Since the documentation seems a bit scarce I am going to throw together some thoughts on how to implement OAuth security realms. For a primer on general plugin development I suggest having a look at the Jenkins Plugin Tutorial.

jenkins-securityJenkins security is split into two parts. The SecruityRealm controlling authentication of users and the AuthorizationStrategy controlling permissions of the users. These two are plugin description points for the respective functionality in Jenkins’ security.

The important thing to remember is that you can implement one without the other. For example the KDE OAuth plugin only implements a SecurityRealm as we currently have no need for our own AuthorizationStrategy. The Role Strategy plugin on the other hand implements only an AuthorizationStrategy.

To successfully implement a SecurityRealm you will need your realm class which is going to extend SecurityRealm and implement a UserDetailsService (this will actually only be used internally to, among other things, log in a user for API transactions). The SecurityRealm will use an AuthenticationToken to actually manage a session and a UserDetails instance to represent a user entity.

You can find some boilerplate code to outline a primitive realm we could use for OAuth2 in this git repository. Which would get a call-sequence similar to this one upon login request:

  • getLoginUrl (redirects to commenceLogin)
  • doCommenceLogin (redirects to request URI on oauth host)
  • doFinishLogin (gets redirected to by oauth host once authorized; requests access token)

After doFinishLogin the user should be authenticated and logged in. As you will probably notice there is talk of MyAuthToken and MyUser. Sample code for those is also available in the git repository.

They are both not terribly complicated, for the most part they are simply plain old data objects representing a session and a user. It is probably worth mentioning that a GrantedAuthority is approximately equal to the concept of a group membership, so much so that if you add more GrantedAuthorityImpls Jenkins will handle them as groups listed on the user profile and for use in AuthorizationStrategies.


MyAuthToken auth = new MyAuthToken(accessToken);
SecurityContextHolder.getContext().setAuthentication(auth);
SecurityListener.fireAuthenticated(auth.getUser());

And that’s all you need for your SecurityRealm. For the most part your realm will simply create a token “somehow” and then set it as active on the SecurityContextHolder. Once that is done you have an authenticated session at your hand.

For some more inspiration hop on over to my actual plugin’s git repository.

A Very Aptly Ruby API

ruby

At Blue Systems we are running a whole slew of different deb repositories all over the place. Primarily of course for the various CI systems we are working on for Plasma Mobile, Debian, and KDE neon.

A while ago Rohan Garg suggested we should use Aptly, a relatively new repository management system written in Go featuring advanced features such as repository mirroring, publishing to Amazon S3 and a REST API. Since I do like me some REST APIs I was quickly sold on the idea of adopting it as our primary repository system.

Alas, most of our CI rigging is written in Ruby and there was no usable API gem in sight.

Low and behold: I wrote a Ruby gem for Aptly’s REST API. It’s called aptly-api.

It covers all the common repository usage from creating repositories and adding files, to publishing repositories to a publishing location. Let’s look at some code.

For example here we create a repository and publish it as public-name.

require 'aptly'

Aptly.configure do |config|
  config.host = 'localhost'
  config.port = 8080
end

repo = Aptly::Repository.create('kewl-new-repo')
repo.publish('public-name', Distribution: 'wily', Architectures: %w(amd64 i386))

Aptly has two notions of a repository. There is a repository and then there is a published repository. A published repository essentially is an on-disk representation of a repository, namely what you would actually find on any deb repository like archive.ubuntu.com. These published repositories have one (or more!) repositories associated with it. All packages in that repository will be part of the published repository. This allows you to have neat repository sets where internally you have packages separate in multiple repositories but publicly they show up in the same published repository tree. For instance you could have a repository for qt and a repository for kde-frameworks, but publicly your deb repository would contain both qt’s and kde-frameworks’ packages.

Let’s add a deb to our repository…

repo = Aptly::Repository.get('kewl-new-repo')
repo.upload('kitteh.deb')

At this point kitteh will not actually show up in the published directory, we first have to update it. In fact, let’s update all published repositories our repository is part of, chances are we want all to refresh anyway.

repo.published_in.each(&:update!)

For more useful information on aptly-api have a look at the gem documentation. If you find aptly-api useful, drop me a comment. Pull requests and bug reports are of course appreciated as well.


Bonus: Secure Remote

Aptly presently doesn’t support API authentication, so you’d rightfully wonder how exactly we secure the beast. Surely we aren’t publicly binding the API server for all the world to abuse.

The solution, as so often in life, is of course SSH. Namely if you want to use the API on a remote you could simply tunnel the relevant port to your localhost.

require 'aptly'
require 'net/ssh/gateway'

gateway = Net::SSH::Gateway.new('hostname', 'username')
port = gateway.open('localhost', 8080, 8080)

Aptly.configure do |config|
  config.host = 'localhost'
  config.port = port
end