User Managed Services

Recently the need occurred for us to run API services from user accounts rather than elevated access (i.e. root). I have since come to like this rather a lot as systemd makes this super easy and in the long run allows more self-management on regular user accounts that need to run daemon services. This is fairly ideal for unprivileged micro services run on shared servers. The basic idea is that every user can run their own systemd services and therefore every user can operate a daemon (if allowed to).

Setting this up initially has some pitfalls though, so I thought I would write down how this is best made to work.


First things first. To make use of this you need systemd, logind and journald. Additionally you’ll need pam_systemd and it needs to be loaded for sessions (distributions will usually set this up automatically for you, if not have fun editing /etc/pam.d/ ;)).


We will also need the actual systemd service/unit file. Generally, everything is the same as if you were to write a regular system service. Ultimately this also means that you can use the same service file for system-wide use or user-limited use so long as the actual service doesn’t require elevated permissions for anything.

A simple example could be this:




Of note is the install target which will enable our service to be started by the default target (i.e. this service would get auto-started on boot).


Before we can get started some additional settings are needed

  1. Enable lingering for the user. This allows user services to exist outside active logind sessions, consequently this needs to be done for any new user which should be able to do this.
    loginctl enable-linger $USERNAME
  2. Enable persistent journald logging. This is optional but without it users are not able to read their own logs unless in the systemd-journal system group.
    mkdir /var/log/journal && systemctl restart systemd-journald
  3. Re-log on the lingering user to make sure permissions are properly applied etc.


To install the service file you’ll want to place it in the home-directory-bound XDG directory as described in the systemd.unit manpage. Usually this would be


Once you placed your .service file in there you’ll probably need to reload the daemon to get it to reload the file systemctl --user daemon-reload


Once everything is configured and installed we can get rocking by running the commands as the user itself.

Start the service with systemctl --user start statifier.service

Verify it started properly with systemctl --user status statifier.service

Enable the service for autostart via target with systemctl --user enable statifier.service

Look at the logs with journalctl  --user statifier.service


By putting everything together you can deploy new code or changes to the service file via sftp and reload and restart the service via ssh systemctl. Allowing for really simple deployment code and no sysadmin involvement beyond the initial setup. And thanks to journald you don’t have to worry about logging since it will gobble up all output and know it came from this service.

I for one love it!

Zabbix IRC Notifications


Some months ago I rolled out the terrifyingly fancy monitoring platform Zabbix to monitor all Blue Systems servers conveniently. Ever since then I wanted IRC notifications but there didn’t seem to be anything compelling available, so I got quickly annoyed and moved on.

Eventually our very own Bhushan Shah poked me enough to figure out IRC notifications.

So, now we have zabbix-irc-pusher. It is an incredibly simple script connecting to IRC and sending messages to a channel. It does so without actually demonizing, which some might argue makes the script simpler. It does however mean that the script will make numerous join/quit messages appear in the relevant IRC channels, so it is advisable to enable outside messages for that channel so the bot doesn’t actually need to join the channel.

Setting the notifications up is a bit meh though, so here’s how. This is talking about Zabbix 2.x, but all of this should largely be the same for the recently released Zabbix 3.x.

First things first. Zabbix has built-in script support that is meant to be a simple notification solution where a specific notification script is simply called with 3 arguments corresponding to an e-mail’s To, Subject and Body field. These notification scripts need to be placed into a directory your zabbix-server uses for alert scripts. You can check the zabbix_server.conf’s AlertScriptsPath variable to find or change the directory in question. By default it will be something like /usr/lib/zabbix/alertscripts/ so we are going to roll with this for now. The script in question needs to be in that directory and made executable. Once the script is working and in the correct directory all the rest of the configuration happens in Zabbix itself.

In Administration→Media types create a new media type, make it type Script and write the name of the script file.


Next you need to use the script as notification strategy for a specific user. Notifications will not be issued if your script is not actually used for notifications on any user!

Go to Administration→Users pick any enabled user and go to the Media tab. Add a new media, select your IRC notification media, set an IRC channel to send notifications to and pick the notifications that should be sent. Don’t forget to actually update the user, once you add the media.


At this point we have the notification method set up, but not the content. To do that we’ll have to configure an action. In Configuration→Actions create a new action and define content.

We use the following:

Name: Report problems to IRC
Default subject: {TRIGGER.STATUS}: {TRIGGER.NAME} -{TRIGGER.ID}&eventid={EVENT.ID}
Default message:
Recovery subject: {TRIGGER.STATUS}: {TRIGGER.NAME} -{TRIGGER.ID}&eventid={EVENT.ID}
Recovery message:

You can define a bunch of conditions in which to notify.

Last but not least, you need to associate the action with the notification method we set up. In the operations tab add a new operation and associate with the user for which you set up the notification method. Don’t forget to actually hit add for the operation and also for the action to save both.

zabbix-irc-04 zabbix-irc-05

Once you are done you should have working IRC notifications. To check simply cause an event (e.g. take an agent offline) and check the event info under Monitoring→Events. Events fitting the action conditions should now have a message actions entry with information about the message delivery and the notification should have arrived on IRC. That’s it!

zabbix-irc-06 zabbix-irc-07

Naturally, all this applies to any script based notification, so whether your script forwards the information to IRC, Telegram or perhaps even an issue tracking system doesn’t really matter as far as the Zabbix side is concerned.

Unfortunately debugging script notifications is a bit of a crafty topic, so to make sure you don’t forget anything here’s a short list of things to do:

  1. Make sure Zabbix-Server has an alert scripts path set up
  2. Put script in alert scripts path
  3. Make script exectuable (chmod +x)
  4. In Zabbix add a media with type script and the relevant script’s file name
  5. Add a notification method to an enabled Zabbix user
  6. Add an action and associate it with the Zabbix user
  7. Check that new events have a message actions entry for the new action