User Managed Services

Recently the need occurred for us to run API services from user accounts rather than elevated access (i.e. root). I have since come to like this rather a lot as systemd makes this super easy and in the long run allows more self-management on regular user accounts that need to run daemon services. This is fairly ideal for unprivileged micro services run on shared servers. The basic idea is that every user can run their own systemd services and therefore every user can operate a daemon (if allowed to).

Setting this up initially has some pitfalls though, so I thought I would write down how this is best made to work.

Dependencies

First things first. To make use of this you need systemd, logind and journald. Additionally you’ll need pam_systemd and it needs to be loaded for sessions (distributions will usually set this up automatically for you, if not have fun editing /etc/pam.d/ ;)).

Unit

We will also need the actual systemd service/unit file. Generally, everything is the same as if you were to write a regular system service. Ultimately this also means that you can use the same service file for system-wide use or user-limited use so long as the actual service doesn’t require elevated permissions for anything.

A simple example could be this:

[Unit]
Description=Statifier

[Service]
Environment=PORT=9000
ExecStart=/home/statifier/bin/statifier
Restart=always

[Install]
WantedBy=default.target

Of note is the install target which will enable our service to be started by the default target (i.e. this service would get auto-started on boot).

Configuration

Before we can get started some additional settings are needed

  1. Enable lingering for the user. This allows user services to exist outside active logind sessions, consequently this needs to be done for any new user which should be able to do this.
    loginctl enable-linger $USERNAME
  2. Enable persistent journald logging. This is optional but without it users are not able to read their own logs unless in the systemd-journal system group.
    mkdir /var/log/journal && systemctl restart systemd-journald
  3. Re-log on the lingering user to make sure permissions are properly applied etc.

Installation

To install the service file you’ll want to place it in the home-directory-bound XDG directory as described in the systemd.unit manpage. Usually this would be

~/.config/systemd/user/

Once you placed your .service file in there you’ll probably need to reload the daemon to get it to reload the file systemctl --user daemon-reload

Running

Once everything is configured and installed we can get rocking by running the commands as the user itself.

Start the service with systemctl --user start statifier.service

Verify it started properly with systemctl --user status statifier.service

Enable the service for autostart via target with systemctl --user enable statifier.service

Look at the logs with journalctl  --user statifier.service

Conclusion

By putting everything together you can deploy new code or changes to the service file via sftp and reload and restart the service via ssh systemctl. Allowing for really simple deployment code and no sysadmin involvement beyond the initial setup. And thanks to journald you don’t have to worry about logging since it will gobble up all output and know it came from this service.

I for one love it!

Zabbix IRC Notifications

Zabbix_logo

Some months ago I rolled out the terrifyingly fancy monitoring platform Zabbix to monitor all Blue Systems servers conveniently. Ever since then I wanted IRC notifications but there didn’t seem to be anything compelling available, so I got quickly annoyed and moved on.

Eventually our very own Bhushan Shah poked me enough to figure out IRC notifications.

So, now we have zabbix-irc-pusher. It is an incredibly simple script connecting to IRC and sending messages to a channel. It does so without actually demonizing, which some might argue makes the script simpler. It does however mean that the script will make numerous join/quit messages appear in the relevant IRC channels, so it is advisable to enable outside messages for that channel so the bot doesn’t actually need to join the channel.

Setting the notifications up is a bit meh though, so here’s how. This is talking about Zabbix 2.x, but all of this should largely be the same for the recently released Zabbix 3.x.

First things first. Zabbix has built-in script support that is meant to be a simple notification solution where a specific notification script is simply called with 3 arguments corresponding to an e-mail’s To, Subject and Body field. These notification scripts need to be placed into a directory your zabbix-server uses for alert scripts. You can check the zabbix_server.conf’s AlertScriptsPath variable to find or change the directory in question. By default it will be something like /usr/lib/zabbix/alertscripts/ so we are going to roll with this for now. The script in question needs to be in that directory and made executable. Once the script is working and in the correct directory all the rest of the configuration happens in Zabbix itself.

In Administration→Media types create a new media type, make it type Script and write the name of the script file.

zabbix-irc-01

Next you need to use the script as notification strategy for a specific user. Notifications will not be issued if your script is not actually used for notifications on any user!

Go to Administration→Users pick any enabled user and go to the Media tab. Add a new media, select your IRC notification media, set an IRC channel to send notifications to and pick the notifications that should be sent. Don’t forget to actually update the user, once you add the media.

zabbix-irc-02

At this point we have the notification method set up, but not the content. To do that we’ll have to configure an action. In Configuration→Actions create a new action and define content.

We use the following:

Name: Report problems to IRC
Default subject: {TRIGGER.STATUS}: {TRIGGER.NAME} - http://m.neon.kde.org/zabbix/tr_events.php?triggerid={TRIGGER.ID}&eventid={EVENT.ID}
Default message:
Recovery subject: {TRIGGER.STATUS}: {TRIGGER.NAME} - http://m.neon.kde.org/zabbix/tr_events.php?triggerid={TRIGGER.ID}&eventid={EVENT.ID}
Recovery message:

You can define a bunch of conditions in which to notify.

Last but not least, you need to associate the action with the notification method we set up. In the operations tab add a new operation and associate with the user for which you set up the notification method. Don’t forget to actually hit add for the operation and also for the action to save both.

zabbix-irc-04 zabbix-irc-05

Once you are done you should have working IRC notifications. To check simply cause an event (e.g. take an agent offline) and check the event info under Monitoring→Events. Events fitting the action conditions should now have a message actions entry with information about the message delivery and the notification should have arrived on IRC. That’s it!

zabbix-irc-06 zabbix-irc-07

Naturally, all this applies to any script based notification, so whether your script forwards the information to IRC, Telegram or perhaps even an issue tracking system doesn’t really matter as far as the Zabbix side is concerned.

Unfortunately debugging script notifications is a bit of a crafty topic, so to make sure you don’t forget anything here’s a short list of things to do:

  1. Make sure Zabbix-Server has an alert scripts path set up
  2. Put script in alert scripts path
  3. Make script exectuable (chmod +x)
  4. In Zabbix add a media with type script and the relevant script’s file name
  5. Add a notification method to an enabled Zabbix user
  6. Add an action and associate it with the Zabbix user
  7. Check that new events have a message actions entry for the new action

 

Building a Jenkins Security Realm

java

Last week I spent a good while on writing a new security realm for KDE’s Jenkins setups. The result of my tireless java brewing is that the Jenkins installation of KDE neon now uses KDE’s Phabricator setup to authenticate users and manage permissions via OAuth.

We should hopefully see this roll out to the KDE CI Jenkins as well in the near future.

Since the documentation seems a bit scarce I am going to throw together some thoughts on how to implement OAuth security realms. For a primer on general plugin development I suggest having a look at the Jenkins Plugin Tutorial.

jenkins-securityJenkins security is split into two parts. The SecruityRealm controlling authentication of users and the AuthorizationStrategy controlling permissions of the users. These two are plugin description points for the respective functionality in Jenkins’ security.

The important thing to remember is that you can implement one without the other. For example the KDE OAuth plugin only implements a SecurityRealm as we currently have no need for our own AuthorizationStrategy. The Role Strategy plugin on the other hand implements only an AuthorizationStrategy.

To successfully implement a SecurityRealm you will need your realm class which is going to extend SecurityRealm and implement a UserDetailsService (this will actually only be used internally to, among other things, log in a user for API transactions). The SecurityRealm will use an AuthenticationToken to actually manage a session and a UserDetails instance to represent a user entity.

You can find some boilerplate code to outline a primitive realm we could use for OAuth2 in this git repository. Which would get a call-sequence similar to this one upon login request:

  • getLoginUrl (redirects to commenceLogin)
  • doCommenceLogin (redirects to request URI on oauth host)
  • doFinishLogin (gets redirected to by oauth host once authorized; requests access token)

After doFinishLogin the user should be authenticated and logged in. As you will probably notice there is talk of MyAuthToken and MyUser. Sample code for those is also available in the git repository.

They are both not terribly complicated, for the most part they are simply plain old data objects representing a session and a user. It is probably worth mentioning that a GrantedAuthority is approximately equal to the concept of a group membership, so much so that if you add more GrantedAuthorityImpls Jenkins will handle them as groups listed on the user profile and for use in AuthorizationStrategies.


MyAuthToken auth = new MyAuthToken(accessToken);
SecurityContextHolder.getContext().setAuthentication(auth);
SecurityListener.fireAuthenticated(auth.getUser());

And that’s all you need for your SecurityRealm. For the most part your realm will simply create a token “somehow” and then set it as active on the SecurityContextHolder. Once that is done you have an authenticated session at your hand.

For some more inspiration hop on over to my actual plugin’s git repository.