DrKonqi ❀️ coredumpd

Get some popcorn and strap in for a long one! I shall delight you with some insights into crash handling and all that unicorn sparkle material.


Since Plasma 5.24 DrKonqi, Plasma’s infamous crash reporter, has gained support to route crashes through coredumpd and it is amazing – albeit a bit unused. That is why I’m telling you about it now because it’s matured a bit and is even more amazing – albeit still unused, I hope that will change.

To explain what any of this does I have to explain some basics first, so we are on the same page…

Most applications made by KDE will generally rely on KCrash, a KDE framework that implements crash handling, to, well, handle crashes. The way this works depends a bit on the operating system but one way or another when an application encounters a fault it first stops to think for a moment, about the meaning of life and whatever else, we call that “catching the crash”, during that time frame we can apply further diagnostics to help later figure out what went wrong. On POSIX systems specifically, we generate a backtrace and send that off to our bugzilla for handling by a developer – that is in essence the job of DrKonqi.

Currently DrKonqi operates in a mode of operation generally dubbed “just-in-time debugging”. When a crash occurs: KCrash immediately starts DrKonqi, DrKonqi attaches GDB to the still running process, GDB creates a backtrace, and then DrKonqi sends the trace along with metadata to bugzilla.

Just-in-time debugging is often useful on developer machines because you can easily switch to interactive debugging and also have a more complete picture of the environmental system state. For user systems it is a bit awkward though. You may not have time to deal with the report right now, you may have no internet connection, indeed the crash may be impossible to trace because of technical complications occurring during just-in-time debugging because of how POSIX signals work (threads continue running :O), etc.

In short: just-in-time really shouldn’t be the default.

Enter coredumpd.

Coredumpd is part of systemd and acts as kernel core handler. Ah, that’s a mouthful again. Let’s backtrace (pun intended)… earlier when I was talking about KCrash I only told part of the story. When fault occurs it doesn’t necessarily mean that the application has to crash, it could also neatly exit. It is only when the application takes no further action to alleviate the problem that the Linux kernel will jump in and do some rudimentary crash handling, forcefully. Very rudimentary indeed, it simply takes the memory state of the process and dumps it into a file. This is then aptly called a core dump. It’s kind of like a snapshot of the state of the process when the fault occurred and allows for debugging after the fact. Now things get interesting, don’t they? πŸ™‚

So… KCrash can simply do nothing and let the Linux kernel do the work, and the Linux kernel can also be lazy and delegate the work to a so called core handler, an application that handles the core dumping. Well, here we are. That core handler can be coredumpd, making it the effective crash handler.

What’s the point you ask? — We get to be lazy!

Also, core dumping has one huge advantage that also is its disadvantage (depending on how you look at it): when a core dumps, the process is no longer running. When backtracing a core dump you are looking at a snapshot of the past, not a still running process. That means you can deal with crashes now or in 5 minutes or in 10 hours. So long as the core dump is available on disk you can trace the cause of the crash. This is further improved by coredumpd also storing a whole lot of metadata in journald. All put together it allows us to run drkonqi after-the-fact, instead of just-in-time. Amazing! I’m sure you will agree.

For the user everything looks the same, but under the hood we’ve gotten rid of various race conditions and gotten crash persistence across reboots for free!

Among other things this gives us the ability to look at past crashes. A GUI for which will be included in Plasma 5.25. Future plans also include the ability to file bug reports long after the fact.

Inner Workings

The way this works behind the scenes is somewhat complicated but should be easy enough to follow:

  • The application produces a fault
  • KCrash writes KCrash-specific metadata into a file on disk and doesn’t exit
  • The kernel issues a core dump via coredumpd
  • The systemd unit coredump@ starts
  • At the same time drkonqi-coredump-processor@ starts
  • The processor@ waits for coredump@ to finishes its task of dumping the core
  • The processor@ starts drkonqi-coredump-launcher@ in user scope
  • launcher@ starts DrKonqi with the same arguments as though it had been started just-in-time
  • DrKonqi assembles all the data to produce a crash report
  • the user is greeted by a crash notification just like just-in-time debugging
  • the entire crash reporting procedure is the same

Use It!

If you are using KDE neon unstable edition you are already using coredumpd based crash reporting for months! You haven’t even noticed, have you? πŸ˜‰

If not, here’s your chance to join the after-the-fact club of cool kids.

KCRASH_DUMP_ONLY=1

in your `/etc/environment` and make sure your distribution has enabled the relevant systemd units accordingly.

Reuse Licensing Helper

It’s boring but important! Stay with me! Please! 😘

For the past couple of years Andreas Cord-Landwehr has done excellent work on moving KDE in a more structured licensing direction. Free software licensing is an often overlooked topic, that is collectively understood to be important, but also incredibly annoying, bureaucratic, and complex. We all like to ignore it more than we should.

If you are working on KDE software you really should check out KDE’s licenses howto and maybe also glance over the comprehensive policy. In particular when you start a new repo!

I’d like to shine some light on a simple but incredibly useful tool: reuse. reuse helps you check licensing compliance with some incredibly easy commands.

Say you start a new project. You create your prototype source, maybe add a readme – after a while it’s good enough to make public and maybe propose for inclusion as mature KDE software by going through KDE Review. You submit it for review and if you are particularly unlucky you’ll have me come around the corner and lament how your beautiful piece of software isn’t completely free software because some files lack any sort of licensing information. Alas!

See, you had better use reuse…

pip3 install --user reuse

reuse lint: lints the source and tells you which files aren’t licensed

reuse download --all: downloads the complete license files needed for compliance based on the licenses used in your source (unfortunately you’ll still need to manually create the KDEeV variants)

If you are unsure how to license a given file, consult the licensing guide or the policy or send a mail to one of the devel mailing lists. There’s help a plenty.

Now that you know about the reuse tool there’s even less reason to start projects without 100% compliance so I can shut up about it πŸ™‚

No SMB1 to Share Devices

As it recently came up I thought I should perhaps post this more publicly…

As many of you will know SMB1, the super ancient protocol for windows shares, shouldn’t be used anymore. It’s been deprecated since like Windows Vista and was eventually also disabled by default in both Windows 10 and Samba. As a result you are not able to find servers that do not support either DNS-SD aka Bonjour aka Avahi, or WS-Discovery. But there’s an additional problem! Many devices (e.g. NAS) produced since the release of Vista could support newer versions of SMB but for not entirely obvious reasons do not have support for WS-Discovery-based … discovery. So, you could totally find and use a device without having to resort to SMB1 if you know its IP address. But who wants to remember IP addresses.

Instead you can just have another device on the network play discovery proxy! One of the many ARM boards out there, like a rapsberrypi, would do the trick.

To publish a device over DNS-SD (for Linux & OS X) you’ll first want to map its IP address to a local hostname and then publish a service on that hostname.

avahi-publish -a blackbox.local 192.168.100.16
avahi-publish -s -H blackbox.local SMB_ON_BLACKBOX _smb._tcp 445

If you also want to publish for windows 10 systems you’ll additionally want to run wsdd

wsdd.py -v -n BLACKBOX

Do note that BLACKBOX in this case can be a netbios, or LLMNR, or DNS-SD name (Windows 10 does support name resolution via DNS-SD these days). Unfortunate caveat of wsdd is that if you want to publish multiple devices you’ll need to set up a bridge and put the different wsdd instances on different interfaces.