Aptly Systemd Socket Activation

In the previous blog post I talked about Unix domain socket enablement in Aptly, a very popular deb repository management solution. In addition to adding socket support, I’ve also added support for systemd socket activation.

Systemd’s socket activation is an inordinately fancy feature of systemd where it will take over socket creation and management before the actual service is running. It ensures that incoming requests aren’t lost if the service crashes or isn’t started yet. When one has the opportunity to use systemd, letting it do socket activation for Aptly is super handy as it also gives easy access to chown capabilities.

Using socket activation requires you to create two systemd units. A service unit, describing the aptly service itself, and a socket unit, for listening for traffic. Examples files may look like this:

# aptly.service
[Unit]
Description=Aptly archive service

[Service]
ExecStart=/home/archive/bin/aptly api serve
WorkingDirectory=/home/archive
Restart=always
# aptly.socket
[Unit]
Description=Aptly archive service socket

[Socket]
ListenStream=/home/archive/aptly.sock
SocketMode=0660
SocketUser=archive
SocketGroup=archivesocket

[Install]
WantedBy=sockets.target

Enabling and starting the socket unit will create the socket file at /home/archive/aptly.sock and start the service upon incoming traffic, which you could cause, for example, using curl --unix-socket /home/archive/aptly.sock http:/api/version.

Fairly exciting already. But if you’ve previously been listening on a TCP port you may wish to establish a smoother migration path without having to change your entire infrastructure to work with sockets all at once. Aptly however only supports listening on one socket, so you can’t just have it listen on both a TCP socket and a unix socket. Fortunately, systemd also has a neat solution for this, called systemd-socket-proxyd. Using the socket activation systemd itself can proxy traffic from a TCP socket to our new aptly socket file.

Again example units for illustration:

# aptly-proxy.service
[Unit]
Description=Aptly Socket (Proxy Listener)

[Service]
ExecStart=/lib/systemd/systemd-socket-proxyd /home/archive/aptly.sock
PrivateTmp=yes
# aptly-proxy.socket
[Unit]
Description=Aptly Socket (Proxy Listener)
After=aptly.socket

[Socket]
ListenStream=127.0.0.1:8080

[Install]
WantedBy=sockets.target

When enabling and starting this socket unit systemd will listen on 127.0.0.1:8080 for traffic and proxy it through systemd-socket-proxyd to our actual Aptly socket file. At this point Aptly will be able to receive traffic from both the unix socket and the TCP socket (although of course effectively Aptly is only listening on the unix socket).

Now if only systemd could play audio it’d be quite the bundle of usefulness 😉

User Managed Services

Recently the need occurred for us to run API services from user accounts rather than elevated access (i.e. root). I have since come to like this rather a lot as systemd makes this super easy and in the long run allows more self-management on regular user accounts that need to run daemon services. This is fairly ideal for unprivileged micro services run on shared servers. The basic idea is that every user can run their own systemd services and therefore every user can operate a daemon (if allowed to).

Setting this up initially has some pitfalls though, so I thought I would write down how this is best made to work.

Dependencies

First things first. To make use of this you need systemd, logind and journald. Additionally you’ll need pam_systemd and it needs to be loaded for sessions (distributions will usually set this up automatically for you, if not have fun editing /etc/pam.d/ ;)).

Unit

We will also need the actual systemd service/unit file. Generally, everything is the same as if you were to write a regular system service. Ultimately this also means that you can use the same service file for system-wide use or user-limited use so long as the actual service doesn’t require elevated permissions for anything.

A simple example could be this:

[Unit]
Description=Statifier

[Service]
Environment=PORT=9000
ExecStart=/home/statifier/bin/statifier
Restart=always

[Install]
WantedBy=default.target

Of note is the install target which will enable our service to be started by the default target (i.e. this service would get auto-started on boot).

Configuration

Before we can get started some additional settings are needed

  1. Enable lingering for the user. This allows user services to exist outside active logind sessions, consequently this needs to be done for any new user which should be able to do this.
    loginctl enable-linger $USERNAME
  2. Enable persistent journald logging. This is optional but without it users are not able to read their own logs unless in the systemd-journal system group.
    mkdir /var/log/journal && systemctl restart systemd-journald
  3. Re-log on the lingering user to make sure permissions are properly applied etc.

Installation

To install the service file you’ll want to place it in the home-directory-bound XDG directory as described in the systemd.unit manpage. Usually this would be

~/.config/systemd/user/

Once you placed your .service file in there you’ll probably need to reload the daemon to get it to reload the file systemctl --user daemon-reload

Running

Once everything is configured and installed we can get rocking by running the commands as the user itself.

Start the service with systemctl --user start statifier.service

Verify it started properly with systemctl --user status statifier.service

Enable the service for autostart via target with systemctl --user enable statifier.service

Look at the logs with journalctl  --user statifier.service

Conclusion

By putting everything together you can deploy new code or changes to the service file via sftp and reload and restart the service via ssh systemctl. Allowing for really simple deployment code and no sysadmin involvement beyond the initial setup. And thanks to journald you don’t have to worry about logging since it will gobble up all output and know it came from this service.

I for one love it!