Systemd as tragedy [LWN.net]

Systemd as tragedy

Posted Jan 30, 2019 4:25 UTC (Wed) by j16sdiz (guest, #57302)
In reply to: Systemd as tragedy by rahulsundaram
Parent article: Systemd as tragedy

Systemd does make common use case very smooth.
For corner cases, it break your hack every few versions.

Systemd as tragedy

Posted Jan 30, 2019 11:15 UTC (Wed) by Limax (guest, #129516) [Link]

I am curious. What corner cases do you have?

Systemd as tragedy

Posted Feb 5, 2019 0:17 UTC (Tue) by jccleaver (guest, #127418) [Link] (22 responses)

> Systemd does make common use case very smooth.

HAHAHAHA

From: https://www.freedesktop.org/wiki/Software/systemd/Predict...
> Does this have any drawbacks? Yes, it does. Previously it was practically guaranteed that hosts equipped with a single ethernet card only had a single "eth0" interface. With this new scheme in place, an administrator now has to check first what the local interface name is before they can invoke commands on it where previously they had a good chance that "eth0" was the right name.

Systemd as tragedy

Posted Feb 5, 2019 1:28 UTC (Tue) by anselm (subscriber, #2796) [Link] (21 responses)

And please, sir, how is that a systemd problem?

Systemd as tragedy

Posted Feb 5, 2019 1:44 UTC (Tue) by jccleaver (guest, #127418) [Link] (20 responses)

> And please, sir, how is that a systemd problem?

My thoughts exactly -- but probably not in how you were envisioning the question.

Systemd as tragedy

Posted Feb 5, 2019 2:03 UTC (Tue) by pizza (subscriber, #46) [Link] (19 responses)

udev has supported interface renaming since approximately forever, well before systemd ever existed, because no matter what an interface is named, having it get a different name on the next bootup (due to device enumeration being highly non-deterministic) is a recipe for disaster.

(I'm personally a big fan of the 'biosdevname' policy. Because why shouldn't default device names match the vendor's labels on the physical ports?)

Systemd as tragedy

Posted Feb 5, 2019 8:45 UTC (Tue) by jccleaver (guest, #127418) [Link] (18 responses)

> udev has supported interface renaming since approximately forever, well before systemd ever existed, because no matter what an interface is named, having it get a different name on the next bootup (due to device enumeration being highly non-deterministic) is a recipe for disaster.

That's great, and this no doubt solves a corner case for someone. But having worked at massive Dell shops I can say that even on 6-NIC servers I've *never* been hit by spontaneous random interface re-ordering on boot. Ever. Furthermore, the simplest possible case, *especially* in a VM-focused environment, is undoubtedly a "single host with single NIC", and whatever benefits un-"predictable device naming" purports to provide is completely obliterated by no longer being able to assume eth0 has a meaning, when 99.5% of the time it previously worked as expected. (The only issue addressed here is when your MAC changes, but all major hypervisors have hooks in guest services to deal with that.)

Requiring others to re-code, re-design, or insert hacks into software and configs to solve weird edge cases LP found on his laptop one day and then telling everyone they should be happy in the long run for this extra work epitomizes the systemd "cabal"'s approach to doing things.

Systemd as tragedy

Posted Feb 5, 2019 9:46 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

> That's great, and this no doubt solves a corner case for someone. But having worked at massive Dell shops I can say that even on 6-NIC servers I've *never* been hit by spontaneous random interface re-ordering on boot. Ever.
I have that happened on my desktop with two Ethernet ports.

Systemd as tragedy

Posted Feb 5, 2019 13:05 UTC (Tue) by pizza (subscriber, #46) [Link] (15 responses)

Are you sure that isn't because udev has hidden re-ordering behind its default behaviour of persistent interface renaming?

Systemd as tragedy

Posted Feb 5, 2019 15:28 UTC (Tue) by jccleaver (guest, #127418) [Link] (13 responses)

> > That's great, and this no doubt solves a corner case for someone. But having worked at massive Dell shops I can say that even on 6-NIC servers I've *never* been hit by spontaneous random interface re-ordering on boot. Ever.

> Are you sure that isn't because udev has hidden re-ordering behind its default behaviour of persistent interface renaming?

Pretty sure, unless the times we did that intentionally were logged in a different way than when it's automatically performing the function. That also really only helps for persistency once the OS is installed -- if it were truly random, then kickstarts without MAC codification would be expected to randomly fail as NICs went to different switches or ports got bonded incorrectly, and that didn't happen for us with any of our models, even though only the GB1 NIC was used by us for tracking in our PXE boot server.

Of course, the persistence with udev kind of proves the point: Even if the kernel can't be trusted deterministically, and with biosdevname out there, udev was already taking care of it. It was a "solved problem". There was no need for systemd to rip everything out and remove the long-standing conventions and make the most common case (a box with a single nic) annoying and difficult.

And with udev being subsumed into systemd, for no good reason, no one outside of the project had any real say in the matter.

Systemd as tragedy

Posted Feb 6, 2019 0:00 UTC (Wed) by anselm (subscriber, #2796) [Link] (12 responses)

And with udev being subsumed into systemd, for no good reason, no one outside of the project had any real say in the matter.

The principal maintainers of udev also happen to be the principal maintainers of systemd, and they seem to find it convenient to arrange things this way. According to them it avoids code duplication and cyclic dependencies and makes administering the code base easier. Their prerogative.

Also, AFAIR the connection between systemd and udev doesn't really go deeper than the fact that they're sharing the same upstream tarball. It is still possible to build and use udev without systemd (and presumably vice-versa, although there would probably be little point – on the whole, udev for its faults seems to be a lot less controversial than systemd).

In any case, the network-interface naming issue is strictly a udev thing and has nothing whatsoever to do with systemd. It also applies to non-systemd installations if they're using udev, and IIRC even predates systemd altogether.

Systemd/udev and complex simple single nic cases

Posted Feb 6, 2019 0:28 UTC (Wed) by jccleaver (guest, #127418) [Link] (11 responses)

> Also, AFAIR the connection between systemd and udev doesn't really go deeper than the fact that they're sharing the same upstream tarball. It is still possible to build and use udev without systemd (and presumably vice-versa, although there would probably be little point – on the whole, udev for its faults seems to be a lot less controversial than systemd).

But that's not really the case operationally, and it's why eudev was forked away from it for Gentoo. Once you combine and remove independent support, let's face it: systemd may have different binaries, but when you take away any attempt and a stable platform between those pieces, you're forcing use of it as an integrated whole.
From: https://lwn.net/Articles/529314/
> > Earlier this year, udev upstream was absorbed into systemd. udev often breaks compatibility with older systems by depending upon recent Linux kernel releases, even when such dependencies are avoidable. This became worse after udev became part of systemd, which has jeopardized our ability to support existing installations. The systemd developers are uninterested in providing full support in udev to systemd alternatives.

> In any case, the network-interface naming issue is strictly a udev thing and has nothing whatsoever to do with systemd. It also applies to non-systemd installations if they're using udev, and IIRC even predates systemd altogether.

Sure, but it's under the systemd aegis that they decided to implement this change, and with removing udev's support for non-systemd distributions, you no longer really have an option. Rather than allow for distinct approaches, now you're forced (well, by default) to adopt this new methodology that solves for edge cases by frustrating the simple, common case:

From: https://www.freedesktop.org/wiki/Software/systemd/Predict...
> > Starting with v197 systemd/udev will automatically assign predictable, stable network interface names for all local Ethernet, WLAN and WWAN interfaces. This is a departure from the traditional interface naming scheme ("eth0", "eth1", "wlan0", ...), but should fix real problems.

Thanks, Lennart. *slow clap*

Systemd/udev and complex simple single nic cases

Posted Feb 6, 2019 9:48 UTC (Wed) by anselm (subscriber, #2796) [Link]

Earlier this year, udev upstream was absorbed into systemd. udev often breaks compatibility with older systems by depending upon recent Linux kernel releases, even when such dependencies are avoidable. This became worse after udev became part of systemd, which has jeopardized our ability to support existing installations. The systemd developers are uninterested in providing full support in udev to systemd alternatives.

That would be the developers of the “eudev” fork of udev talking, so this is not exactly an unbiased view. Whether that fork was really necessary is by no means clear.

Systemd/udev and complex simple single nic cases

Posted Feb 6, 2019 12:50 UTC (Wed) by pizza (subscriber, #46) [Link] (9 responses)

> https://www.freedesktop.org/wiki/Software/systemd/Predict...
> Thanks, Lennart. *slow clap*

For all your bellyaching, that feature is trivially disabled, as per the second-to-last section on that page.

So, yes, thanks Lennart, for building a mechanism that most distributions found useful enough to enable by default, documenting it heavily, and providing an easy way to configure or disable it altogether.

Systemd/udev and complex simple single nic cases

Posted Feb 6, 2019 18:33 UTC (Wed) by jccleaver (guest, #127418) [Link] (8 responses)

> For all your bellyaching, that feature is trivially disabled, as per the second-to-last section on that page.

I'm aware that it's disableable, and regularly do so. My original point was that "making the common use case very smooth" is risible for systemd as a whole. The most common use case for a host will be a single attached NIC, and now your options for this are:
a) a completely unpredictable in advance interface name
b) add kernel flags at install/kickstart and on your grub line to undo this

And it's not even difficult to come up with workarounds for this -- maybe a built-in alias so that "eth0" *always* means "the only NIC" on single NIC systems even if the real name is its mess, so that thing would continue to work the way they had for the last 20 years in most cases.

> So, yes, thanks Lennart, for building a mechanism that most distributions found useful enough to enable by default, documenting it heavily, and providing an easy way to configure or disable it altogether.

That's a sheer bandwagon argument, and it fails even before you get to the underlying arguments about how responsive some distributions are to either their userbases or their downstreams (hint: not always, especially if they regularly kick policy decisions over to members in the same group popping on a different hat).

Systemd/udev and complex simple single NIC cases

Posted Feb 6, 2019 22:26 UTC (Wed) by jccleaver (guest, #127418) [Link] (7 responses)

And BTW, I'm not alone in this. This is a major annoyance and impacted tons of shops to "fix" a problem that was already basically solved.

https://www.ispcolohost.com/2015/07/11/kickstarting-with-...
https://access.redhat.com/discussions/916973
https://access.redhat.com/discussions/2620221

And of course, this doesn't even address the straight-up clusterf*ck of the 7.2 to 7.3 update:

https://access.redhat.com/solutions/2592561
https://bugzilla.redhat.com/show_bug.cgi?id=1392506
https://bugzilla.redhat.com/show_bug.cgi?id=1391944
https://github.com/systemd/systemd/commit/6c1e69f9456d022...

Nor does it hit the "systemd will silently hang all boot/reboot attempts because no one tested SELinux policy application" that many of us hit while trying to address the issue: https://bugzilla.redhat.com/show_bug.cgi?id=1393505

Although this was a flat-out collapse of RedHat QA, it was precipitated by a belief by the systemd crew that it should be able to barrel over all sorts of underlying real-world behaviors blithely making assumptions no one had guaranteed and no one had agreed to. And when it inevitably breaks because of some other issue out there, their initial response is *always* "Hey, works for us. Not our problem!"

See also: UEFI bricking, usernames starting with a number, incorrect /etc/fstab entries hanging boot, flooding debug data to klogd, and God knows what else...

Systemd/udev and complex simple single NIC cases

Posted Feb 6, 2019 23:00 UTC (Wed) by johannbg (guest, #65743) [Link] (2 responses)

You used Dell right and it rings no bells that your own server manufacturer came up with biosdevname trying to solve the same problem you claim does not exist, but finally got solved in udev. ( and got dealt with in solaris decades ago )

Red Hat has a number to call for their support contract have you tried using it?

I'm pretty sure they can give you the moral support you need and even re-train you.

And you still arent critisizing systemd in an area you could arguably critisize it for but carry on throwing bricks at a dead pony...

Systemd/udev and complex simple single NIC cases

Posted Feb 9, 2019 22:46 UTC (Sat) by nix (subscriber, #2304) [Link] (1 responses)

So... jccleaver points out lots of examples of actual problems breaking things for real people (not just jccleaver), and your response is...

> Red Hat has a number to call for their support contract have you tried using it?
> I'm pretty sure they can give you the moral support you need and even re-train you.

I'm fairly sure this is not argument in good faith on your part. (At the very least, you are entirely ignoring your interlocutor.)

Systemd/udev and complex simple single NIC cases

Posted Feb 10, 2019 10:50 UTC (Sun) by johannbg (guest, #65743) [Link]

My poor communication skills aside, the underlying problem which is very much real despite others who have not be bitten by it, claiming it is not so, is that network interface device probing is not predictable which result in unpredictable enumeration on those device interfaces.
( this problem is not spesific to linux, they are present on bsd and solaris as well ).

This problem has existed long before udev and came to be as can be seen by historic multiple prope and pray workarounds from various upstreams and distribution ( and *nixes) trying to get it right, all the way down to administrators trying to manage that by themselves by stuffing something as simple like ifconfig $foo name $bar in /etc/rc.conf or write more complex, scripts which boiled down more or less finding the mac addr and then run ifconfig $foo on it's output.

Users can disable or hack the udev rule to their liking if they dont agree with it's implementation.

And systemd's networkd interface configuration file arent more complex than so...

/etc/systemd/network/20-wired.network

[Match]
Name=enp1s0

[Network]
Domains=<your search domains>
DNS=<your dns >
DNS=<your dns >
Address= <your ipv4 address )
Address=< your ipv6 address )
Gateway=< your gateway >

And a simple ( fallback ) dhcp configuration

/etc/systemd/network/99-wired-dhcp.network

[Match]
Name=enp1*

[Network]
DHCP=yes

And for the keen administrator eye he will notices that *both* of the networkd network files combined are atleast half the size of an single
ifcfg script.

The fact is udev is not do blame for the device naming enumiration problem nor is systemd network configuration, complex or hard to understand

So all this constant blaming udev, systemd or both for all the worlds problem approach, to me makes little to no sense.

Systemd/udev and complex simple single NIC cases

Posted Feb 13, 2019 8:44 UTC (Wed) by flussence (guest, #85566) [Link] (3 responses)

>See also: UEFI bricking, usernames starting with a number, incorrect /etc/fstab entries hanging boot, flooding debug data to klogd, and God knows what else...
The UEFI bricking vuln wasn't unique to systemd.

(and personally I'd prefer if /sbin/mount would grok GPT partition metadata so that fstab could go away, but that's neither here nor there.)

Systemd/udev and complex simple single NIC cases

Posted Feb 13, 2019 18:14 UTC (Wed) by Wol (subscriber, #4433) [Link] (1 responses)

How is it supposed to know *where*, if at all, to mount the partition? That's a rather important part of fstab which the GPT tells you nothing about. And no, the Windows habit of "mounting" *everything* is not a good solution.

Cheers,
Wol

Systemd/udev and complex simple single NIC cases

Posted Feb 14, 2019 7:25 UTC (Thu) by zdzichu (subscriber, #17118) [Link]

There are few well-known GPT UUID↔mountpoint mappings, covering most uses (single OS on permanent storage):
https://en.wikipedia.org/wiki/GUID_Partition_Table#Partit...

I think they're utilised by systemd-gpt-mount generator.

Systemd/udev and complex simple single NIC cases

Posted Feb 14, 2019 17:51 UTC (Thu) by nix (subscriber, #2304) [Link]

Maybe it would help it go away in *some* cases, but certainly not all. My home server has 52 entries in /etc/fstab (and hundreds more mounts created by generators). Know how many could be replaced with UEFI scanning? One, and that's only because booting *requires* a UEFI partition devoted to it. Everything else is under at least md and LVM and often under more layers (bcache, LUKS, etc), or isn't a local fs at all.

Systemd as tragedy

Posted Mar 5, 2019 7:07 UTC (Tue) by immibis (subscriber, #105511) [Link]

I only have one Wi-Fi interface and one Ethernet interface but I've had them renamed a few times as I've added or removed other expansion cards.
If they'd stuck to wlan0 and eth0 then I wouldn't have had a problem.

Systemd as tragedy

Posted May 21, 2019 5:03 UTC (Tue) by buchanmilne (guest, #42315) [Link]

That's great, and this no doubt solves a corner case for someone. But having worked at massive Dell shops I can say that even on 6-NIC servers I've *never* been hit by spontaneous random interface re-ordering on boot.

You obviously never had NICs with different drivers, where a kernel patch-release or new initrd resulted in the drivers being loaded in a different order and your network configuration being broken (e.g. IPs, VLANs, bonds/teams being on the wrong devices). To be clear, on quad-GbE servers with dual-intel and dual-broadcom (e.g. Sun Galaxy range), it was quite common to see this. The workaround (before biosdevname) was to hard-code the MAC addresses in the interface configuration files, and then you would have to update them if ever the motherboard was replaced ...

Ever. Furthermore, the simplest possible case, *especially* in a VM-focused environment, is undoubtedly a "single host with single NIC",

Speak for your own environments. In mine, Linux is the hypervisor, and you absolutely want biosdevname when you have two or more physical NICs with 1 physical function and 14 (SR-IOV) virtual functions each.

and whatever benefits un-"predictable device naming" purports to provide is completely obliterated by no longer being able to assume eth0 has a meaning, when 99.5% of the time it previously worked as expected. (The only issue addressed here is when your MAC changes, but all major hypervisors have hooks in guest services to deal with that.)

VMWare (in the vSphere 4.x days) training specifically called out Linux as being very difficult to manage changes to vNICs compared to Windows ... and yes, I have experienced it too. biosdevname worked much better on the VMWare VMs than getting an unused eth0 and an unconfigured eth1 after cloning a template to a new VM.

Requiring others to re-code, re-design, or insert hacks into software and configs to solve weird edge cases LP found on his laptop one day and then telling everyone they should be happy in the long run for this extra work epitomizes the systemd "cabal"'s approach to doing things.

As noted previously in the comments, biosdevname was not invented by LP. In fact, it was invented by Dell, pre-systemd, and was deployed in many distros before the systemd project was started (e.g. RHEL6 with upstart had it by default). And if you don't want it, disable it (net.ifnames=0 at boot or one of the other methods).