They did a good job [LWN.net]

They did a good job

Posted Sep 17, 2015 2:11 UTC (Thu) by dskoll (subscriber, #1630)
Parent article: How Debian managed the systemd transition

I've upgraded a number of systems from Wheezy to Jessie and it really was quite painless. The only real difference I've noticed is that the systemd-enabled Jessie machines boot a whole lot faster than they did when they were Wheezy.

to post comments

They did a good job

Posted Sep 17, 2015 5:48 UTC (Thu) by alison (subscriber, #63752) [Link] (2 responses)

Hear, hear: chapeau to Martin and Michael for a job well done. I run the Testing release, and the transition was both trouble-free and relatively fast. The transition from Fedora 14 to 15, when systemd was introduced, was a lot rockier, and is part of the reason, I suspect, why developers and sysadmins so feared systemd.

They did a good job

Posted Sep 17, 2015 9:12 UTC (Thu) by salimma (subscriber, #34460) [Link] (1 responses)

Fedora serving its purpose as a more leading-edge distro then. Which makes it telling that it has not adopted Btrfs yet, but that's another huge topic on its own.

They did a good job

Posted Sep 17, 2015 22:54 UTC (Thu) by zlynx (guest, #2285) [Link]

Fedora does make it easy to set btrfs as your filesystem even if it is not the default.

I've been using btrfs on my Fedora laptop and a small home file, backup and email server since F20. No problems.

They did a good job

Posted Sep 17, 2015 15:46 UTC (Thu) by deater (subscriber, #11746) [Link] (21 responses)

I've had problems, not with the transition, but with systemd making "interesting" decisions after the upgrade.

For example, on my headless network gateway, the drive with the /home partition failed ( / was fine).

Instead of just booting with the /home drive removed, it instead sat there on the console with a fancy and colorful display letting me know that a partition was missing, counting down for a few minutes before in the end just totally giving up and dropping to single-user mode. Without bothering to start packet forwarding first. This was a headless machine, so my network was down until I could drag a monitor and keyboard over to figure out what was going on.

I've had a few other surprises like this. And I guess that's par for the course when transitioning such a big system component over, but I find myself spending a lot of time fighting against systemd in cases where things used to "just work".

They did a good job

Posted Sep 17, 2015 16:30 UTC (Thu) by MarcB (subscriber, #101804) [Link] (18 responses)

This is more or less a simple change of defaults. And it was documented.

Before Systemd the "nofail" flag was the default (more or less - errors were reported on the console but they did not block anything). Now, users have to explicitly set the flag if they think that it is OK if a specific automatic mount fails.

I *very* strongly prefer the new behaviour.

The old behaviour could lead to undefined system behaviour, especially when combined with the fact that folders do not need to be empty when mounting something into them.

We once had long deprecated software versions running on a system when a network mount failed silently. Fortunately, the input format had changed in the meantime, so that the old software failed completely - and complained loudly - instead of just producing formally correct but logically wrong output. The latter could have gone unnoticed for months and would have cost a lot of money. (Of course, it was a mistake to just mount a network filesystem over the previously local installation in the first place, but stuff like that happens.)

They did a good job

Posted Sep 17, 2015 21:15 UTC (Thu) by deater (subscriber, #11746) [Link]

> This is more or less a simple change of defaults. And it was documented.

Documented in the basement behind a sign saying "beware of leopard"?
Oh, I found it, section 5.6.1 of this document: https://www.debian.org/releases/stable/i386/release-notes... which I apparently missed at the time.

I still think it's a fairly major and surprising change, though I can grudgingly see why an upgrade to systemd wouldn't want to update the fstab file to make all mounts nofail for backward compatability reasons.

They did a good job

Posted Sep 18, 2015 1:11 UTC (Fri) by mchapman (subscriber, #66589) [Link] (11 responses)

> This is more or less a simple change of defaults. And it was documented.

To add to this, it could be argued that the default hasn't changed, and that the old behaviour was just plain wrong. The mount(8) manpage documents a "nofail" option, but there is no corresponding "fail" option. This seems to imply that "fail on error" was always intended to be the default behaviour, and that the old pre-systemd initscripts had simply got it wrong.

I don't actually agree with this view -- if the old behaviour is what distros used then the documentation should have been written to say that -- but I can certainly see the logic in it. I do prefer systemd's stricter handling of mount points during boot though.

They did a good job

Posted Sep 18, 2015 2:15 UTC (Fri) by dlang (guest, #313) [Link] (10 responses)

you should read Ingo's comments from the kernel qotw http://lwn.net/Articles/657428/

it's worth going and reading his full post.

This attitude of "well, if we broke it, it was wrong anyway" is part of the reason people distrust the systemd folks so much.

They did a good job

Posted Sep 18, 2015 4:17 UTC (Fri) by mchapman (subscriber, #66589) [Link]

> you should read Ingo's comments from the kernel qotw http://lwn.net/Articles/657428/

I said it was "logical", not that it was right. :-)

They did a good job

Posted Sep 18, 2015 10:41 UTC (Fri) by MarcB (subscriber, #101804) [Link]

There is a significant difference between ABI/API and the internal behaviour of system components. The first is supposed to be stable (or not, depending on the project), the second is only supposed to be documented and it must be sane (much more so than an ABI, where you can - at a cost - work around insanity).

There explicitly is no stability guarantee in the latter case (as evidenced by every major distribution's release notes). If you do a major upgrade - if your distribution supports that at all - you have to read the release notes, understand them and perform any extra steps that apply to your system. This has always been the deal.
(And not only that, you also have to read the changelog of every major library you use out of the distribution's repo, or have very good testing in place - that is the real effort.)

Of course, individual users might have gotten away without doing that in the past. They were lucky.

Just to make this clear: Had Debian made such a change within the life-cycle of a distribution, I would have been really annoyed.
But during a major upgrade I expect things like that. And for me, this particular change is more than welcome. Aborting the boot on errors is the only sane thing to do. The init system cannot know and understand the filesystems and simply ignoring the errors can lead to anything, while erroring out always leads to the same thing: A very clear and very visible error that should be caught by even the most simple monitoring.

They did a good job

Posted Sep 18, 2015 14:01 UTC (Fri) by deater (subscriber, #11746) [Link]

> you should read Ingo's comments from the kernel qotw

I thought that qotw was posted ironically due to the number of times his projects have broken various ABIs.

They did a good job

Posted Sep 18, 2015 19:08 UTC (Fri) by davidstrauss (subscriber, #85867) [Link] (6 responses)

> This attitude of "well, if we broke it, it was wrong anyway" is part of the reason people distrust the systemd folks so much.

systemd may have a default and an opinion, but it was Debian's prior, unusual default that created this divergence. If you're Saab -- with a long history of having the ignition in the center console -- and switch to stock parts, it is not the fault of the stock steering column manufacturer that ignition is now in a different place than you had put it before.

Also, Debian could have chosen to configure the post-upgrade system to carry over "nofail," but they didn't. It's not like systemd forces you to have mounts fail hard. Complain to the Debian maintainers if you don't like the way they're choosing to configure systemd.

They did a good job

Posted Sep 26, 2015 10:22 UTC (Sat) by rleigh (guest, #14622) [Link] (5 responses)

It wasn't an "unusual default". It had behaved this way for ~20 years and was the intended and expected behaviour for a Debian system.

They did a good job

Posted Sep 26, 2015 11:16 UTC (Sat) by zdzichu (subscriber, #17118) [Link] (4 responses)

Could you show where it was documented to behave that way?

They did a good job

Posted Oct 9, 2015 22:49 UTC (Fri) by nix (subscriber, #2304) [Link] (3 responses)

You're saying that only documented behaviour needs to be preserved? That users are expected to read *all* the documentation that applies to every little part of their system, and if they don't, it's their own damn fault for being annoyed when something that turns out not to have been documented randomly changes behaviour and breaks things?

This is a very legalistic (read, unhelpful) way to make a system. It's a way to make a system that, to be blunt, doesn't work very well.

They did a good job

Posted Oct 10, 2015 9:24 UTC (Sat) by zdzichu (subscriber, #17118) [Link] (2 responses)

If it's not documented, how can it be "intended"? How can you judge if the behavior is correct and it is not a bug? See also encrypted swap support in article – it was another bug exposed by systemd.

They did a good job

Posted Oct 10, 2015 9:49 UTC (Sat) by cebewee (guest, #94775) [Link] (1 responses)

A behavior does not need to be "intended" to be relied upon. It suffices if this behavior is consistently exposed. Then a change has a high chance of breaking user expectations. Quoting dlang's comment:

> you should read Ingo's comments from the kernel qotw http://lwn.net/Articles/657428/

Now, there are of course cases where breaking changes are justified (and I won't judge whether this was the case here), but you cannot blame people for being upset when said changes break their system.

They did a good job

Posted Oct 11, 2015 0:15 UTC (Sun) by mathstuf (subscriber, #69389) [Link]

One of systemd's goals is to make a layer for Linux that can be depended upon. Minor differences like this are the "death by 1000 papercuts" that makes cross-distro deployment a pain. Breaking every other distro in the same way Debian has for years is certainly not the better solution here. More docs would have been better. Even better would be some check in the updater about possible semantic changes in /etc/fstab entries.

They did a good job

Posted Sep 19, 2015 14:21 UTC (Sat) by cortana (subscriber, #24596) [Link] (4 responses)

It would have been nice if the upgrade to systemd would have added 'nofail' to entries in /etc/fstab. That way, the old behaviour would be preserved for existing systems, in a way that is easy to disable once you read the release notes and decide you want the new behaviour.

They did a good job

Posted Sep 19, 2015 15:16 UTC (Sat) by mathstuf (subscriber, #69389) [Link] (3 responses)

One of their goals was to make an upgrade the same as a new install. Plus, who reads release notes in detail? Does Debian have a nice section for "breaking changes" and the like?

They did a good job

Posted Sep 19, 2015 18:36 UTC (Sat) by cortana (subscriber, #24596) [Link] (2 responses)

Everyone who upgrades should read the release notes first. They mention this exact issue: http://www.debian.org/releases/stable/amd64/release-notes...

They did a good job

Posted Sep 19, 2015 21:52 UTC (Sat) by mathstuf (subscriber, #69389) [Link]

That is exactly what I was looking for :) . Bonus points if they give it as reading material in the upgrade tool (and ponies if they filter/sort it by packages you have installed that are affected).

They did a good job

Posted Sep 19, 2015 22:04 UTC (Sat) by mathstuf (subscriber, #69389) [Link]

Hmm, this looks interesting:

> Lack of security support for the ecosystem around libv8 and Node.js
"Don't use Debian's packages for deploying node.js-based services."

I wonder how many will ignore this :( .

They did a good job

Posted Sep 18, 2015 14:00 UTC (Fri) by drag (guest, #31333) [Link] (1 responses)

> Instead of just booting with the /home drive removed, it instead sat there on the console with a fancy and colorful display letting me know that a partition was missing,

How is this different then what was life was like before systemd?

This sort of crap is why 'enterprise' machines have things like ILOM and iDrac and before that serial terminal servers.

They did a good job

Posted Sep 19, 2015 0:55 UTC (Sat) by deater (subscriber, #11746) [Link]

> How is this different then what was life was like before systemd?

For the previous 20 years I've run Linux/UNIX, if /home was unable to mount, the system would continue anyway and just do without. You could still log in, but obviously your files wouldn't be there and ssh and login might complain loudly. Then you could remotely do whatever was needed to get /home fixed/restored.

The new behavior unhelpfully just dumps you to a prompt, which is bad if you are headless and depending on being able to ssh in (or even worse, the machine is your network gateway so all of your other systems drop off the net because packet forwarding never gets started).

The machine doesn't have iLOM. A serial console would have helped in this situation but I have a lot of machines and it would be a wiring nightmare for all of them to have serial consoles.

So anyway yes, this is a distribution issue not necessarily an init issue. It's just hard when your 20-year old mental model of how a Linux server behaves is majorly changed without much warning. I'm used to other major OSes playing this game but I've been spoiled by Debian being a bit more sane over the years.