Thu, 19 Mar 2015
Libvirt Dependencies
Welcome to Yenya's rant about software "features". Today we will have look at libvirt in Fedora and its dependencies. But firstly let me show you a funny picture:
Anyway. I teach a seminar on Linux administration, where one of the tasks is to compile and use one's own kernel. The task for the following week is to create a virtual machine. One of my students had an interesting problem with the second task - virsh refused to start his KVM-based virtual machine with the "command timeout" message.
Digging into the issue, we discovered that it works with the distribution kernel, but not with his custom kernel. Then we found that virsh tries to do a RPC call over D-Bus, which then times out, because the D-Bus object in question was not present. This object is supposed to be provided by a daemon called systemd-machined, which describes itself with the following headline:
This is a tiny daemon that tracks locally running Virtual Machines and Containers in various ways.
This is in fact an understatement, with the real situation being that this
daemon is a core part of the virtualization subsystem, and it is not even
possible to start a libvirt-managed guest without it. We have tried to start
the daemon from the command line, but it immediately exited without a meaningful
message anywhere. The only message in the syslogjournal was
that systemd-machined failed to start when the system was booted.
Long story short, my lucky guess was that systemd-machined could have something to do also with containers, and it might have needed a container support in the kernel. After enabling about five namespaces-related kernel config options and booting a recompiled kernel, we were able to start systemd-machined, and only then we managed to start the VM using virsh.
This spaghetti-structured unstraceable mess of interconnected daemons communicating over D-Bus and providing no meaningful error messages, which is masqueraded under a collective name "systemd", makes me sick quite often.
5 replies for this story:
michich wrote:
If libvirt fails hard when it fails to communicate with systemd-machined and at the same time it reports no useful error message, I'd consider that a bug in libvirt. You did not write what the error about starting systemd-machined said. Was it something like "Failed at step NAMESPACE spawning systemd-machined"? In systemd-machined.service you'll find it declares several security features whose implementation relies on namespaces: PrivateTmp=yes PrivateDevices=yes PrivateNetwork=yes ProtectSystem=full ProtectHome=yes If the kernel does not provide them, the service cannot start. There are many ways to shoot oneself in the foot when configuring a custom kernel. After spending quite a lot of time debugging some issues reported in Red Hat Bugzilla where the reporters failed to disclose they were using custom kernels (and on one occastion, when the fact became apparent, the reporter even refused to do a test under a Fedora kernel), I have developed a dislike for custom kernels. If you got to have one, please at least use the distro's config as the starting point and only trim down what you KNOW you don't need ("make localmodconfig" is nice).
Yenya wrote: Re: michich
My rant is about something other. I do not deny that systemd-machined requires special kernel features. What I find "interesting" is that virsh fails to start a KVM-based virtual machine in a situation when systemd-machined is not running. There is no need for virsh to require systemd-machined. As systemd-machined describes itself, it only tracks the VMs, it should not be a hard requirement for them. Moreover, when I do not use containers, my KVM-based VMs should not fail only because I don't have container-related features in my kernel. Also, the exact "solution" you describe is the reason of why even I don't compile my own kernels anymore - with custom kernel, random pieces of the distribution start failing, even though the features they are missing are not strictly needed for my use cases. This is the reason of recent decline in the number of "voluntary" kernel developers: using custom kernels has become increasingly harder and troublesome.
michich wrote:
I don't know about the goals of libvirt developers, so I cannot say whether the hard requirement on machined is justified or not. As I wrote earlier, at the very least it should report a useful error message. But maybe you're right and it should just continue regardless of the error. Still I don't see this in any way as systemd-machined's fault. I disagree with your point "I'm not knowingly using containers => I should not need container-related features in the kernel". Namespaces have a wider use than just for what people think of as containers. In my view it is perfectly acceptable if programs want to use these features for security hardening or whatever. They don't necessarily have to appear to the user as running in containers. Using localmodconfig it's fairly easy to create a custom kernel that's both quite lean and not lacking essential features. So I disagree with your conclusion.
Yenya wrote:
There are several problems: firstly, I would guess the dependency on systemd-machined has probably been added to libvirt by systemd developers. Even after reading the manpage, I fail to see the benefits systemd-machined brings to the libvirt user. There are probably none. Secondly, I am fully aware of benefits of namespaces besides their use for creating containers (having per-user /tmp is one of the most clever tricks, for example). So the implication you are trying to put into my mouth is in fact different: I would say something like "I don't use containers and I don't depend on namespaces for my applications => the applications should not fail just because the namespaces support is not present in the kernel". It is the same way as (at least for now), the system is still working even when booted with selinux=disabled.
michich wrote:
Looking at libvirt.git, support for systemd-machined was added to libvirt by Daniel P. Berrange, who's not a systemd developer. Though no doubt he communicated with Lennart first. From some of the commit messages, it seems to me the intention was not to have a hard dependency. It may be that the observed hard dependency is simply a bug. This leads me to your corrected implication. Having applications always degrade gracefully when faced with missing kernel features sounds like a nice plan, but may be difficult to achieve in practice. I mean QA most likely only test on the standard kernel. Adding the requirement of testing various kernel configurations would quickly lead to exponential explosion.