The "Devuan" Debian fork

Out of curiosity, what was the prevailing controversy the previous time?

Thank you.

Applaud your stance. I have complete confidence in your editorial choices, it's a big part of what makes LWN a great site to subscribe to. If you set the troll flag on someone, I know it's more than justified.

It's the people whose speech is the most poisonous who generally squeal the loudest about "free speech". Such speech as theirs is not value-add, it's value-destroy for LWN, and the less we have of it, the better.

What does it tell us when Jonathan Corbet deletes a comment from LWN?

It does not tell us anything about him or LWN, because they have earned a very high regard from us.

It does tell us everything we need to know about the comment and the commenter.

Thanks for deleting the trolling, Mr Corbet.

Thank you.

libsystemd doesn't require pid 1 to be systemd however. Is this really worth a fork?

It is worth noting that package that depend on libsystemd but not on systemd as pid 1 will probably not use the library at run time. It's there only to allow the binary to be loaded and the program to look for systemd and do something if, and only if, systemd is pid 1. I don't believe this is worth a fork.

So please, if you don't like systemd, help us (Debian) to make a better OS by providing us test and code to let system that don't run systemd run well, but don't split the community.

I always wondered why the "tight coupling with init" issue has not simply been solved by providing two systemd "start"packages: a systemd-init package that makes systemd the init and a system-notinit package that installs a start script for other inits.

The systemd package would depend on these as alternatives. People with any other init would install the second package, thus keep their init as-is but still solve all dependencies higher up.

But of course, if as you say the goal is to not run systemd at all, then this would not help.
However, why then phrase the discussion in terms of init when this is not actually the contended issue?

Thank you for posting this comment. It describes the case exactly. I'd like to add a piece to the puzzle so far.

For some the reason for separating from Debian is that the previous iteration of Lennartware, i.e. Pulseaudio's injection between ALSA and userspace, was an unmitigated disaster that (in gentle terms) brought no increase in performance or functionality to systems that were well-configured with ALSA. Instead it changed e.g. the meaning of the mixer volume level as applied by most applications, breaking UI behind the application's back.

I waited out the pulseaudio storm to no decrease in functionality. The way I achieved this is with about five stanzas in an apt-prefs file that's still necessary to keep pulseaudio the hell out. This is because even years after Pöttering discontinued his involvement, PA continues to hang on to the model of being the only network-capable sound service in town. (In the case of PA's design it makes the other servers its clients, which brings out various implementation discrepancies in the way PA reimplemented the ALSA APIs.)

Then along came that same guy's "init system, device manager, login manager, cron, DNS resolver, console terminal emulator, package manager, kitchen sink, slopbucket, API breaker" project's adoption as dependency in the Debian Gnome packages. Since then it's been worming its way in piece by piece through dependencies in other programs; most recently bsdutils pulls in libsystemd0 which combines the freshly de-modularized group of libsystemd-{journal,daemon,login,...} program-level APIs. (who knows, maybe it looks smaller that way.)

Opinions of course vary and the technical minutes are too involved to quote here, but in my view there is no problem that systemd solves that isn't one of those introduced by the systemd design itself. Out of the possible reasons to have such an architecture in GNU/Linux, systemd doesn't meet utility, necessity, or even practical desirability.

The inclusion of systemd's libraries (and PAM modules, and root-privileged daemons, and...) isn't the problem by itself, of course: a library that implements its APIs in terms of local services that don't exist can't cause much trouble (unless root). Rather these libraries are an indication, a first warning of some program being about to become feature-dependent on an API from an architecture controlled by people who're on public record wrt their intent to e.g. break interfaces and make lock-step upgrades the norm. Having something like systemd-shim replacing Pöttering's systemd doesn't help here: for a well-defined architecture it matters relatively little who completed the implementation (or even how).

There are many people who'd like to sit out systemd as could be done with Pulseaudio. Personally I'd prefer to accomplish this by some means that isn't eternal vigilance towards my preferred distribution's packagers and politicians. Such things weren't needed in the time when Debian was a distribution of GNU/Linux first and one of Gnome only a distant second alongside every other userspace program.

Pinning against the entirety of systemd and packages that depend on it would result in many programs staying behind from a time between Wheezy and Jessie while others moved ahead. This tends to expose unstated dependencies between userspace programs which hadn't been reported by the legion of people upgrading regularly from `testing'. The mutually exclusive options for systemd doubters are therefore 0) adopt systemd and damn the icebergs, 1) pin against systemd and become a dinosaur, and 2) pool efforts, fork Debian, and move on.

Hence Devuan. (which, for the record, I'm not actively involved in.)

If things don't work out for it, there'll be others. Haters won't slow this puppy down.

> Do you have any evidence for your claims?

You sound like you're asking for another systemd bug report or CVE. Sorry, I'm busy doing productive work but if you like systemd go ahead and use it. I'm not trying to stop you.

Software engineers and sysadmins have determined that in their professional judgment systemd is a bad idea - bad design and bad policy and bad politics and bad business. By an interesting variety of routes we are coding around the systemd roadblock.

Nobody disputes that some people want systemd. Nobody is trying to prevent the people who want systemd from using it.

However, there are some people who don't want to use systemd, and they are asking for you to let them.

If Devuan produces a viable distro, then the rest of us benefit from having that distro available.

If Devuan is not so successful, there will (probably) be some useful lessons we can learn from it.

In either case, those of us not involved in Devuan come out ahead. Only the people who work on (or donate to?) Devuan are risking anything. Good on them.

Yeah, likewise. If I'm "resisting" using systemd in Gentoo, I've similarly resisted grub2 for the past 5 years. I expect I'll switch to systemd sometime before 2020, but who knows.

that is the proper way to support systemd. I need to try Gentoo again (I was running it for a while, but hit a problem with my server at a time when there wasn't a good way to bootstrap Gentoo (something that worked prior to that and works again afterwords)

I don't run Gentoo, but my understanding from previous conversions is that Gentoo required systemd in the exact same situation that Debian did: for Gnome. (Of course, Debian now lets you use Gnome without systemd just fine, since you can use systemd-shim, instead, to satisfy Gnome's logind requirement.)

So that doesn't really seem to be a distinguishing characteristic of Gentoo vs Debian, AFAICT?

"How do you know "sysadmins dont't want systemd"?"

I think that the Debian fork in progress should be a good example:
http://debianfork.org/

They describe themselves as "Veteran Unix Admins".
You can assume with a high degree of probability that they know their stuff. And that includes a healthy dose of confidence in their own abilities and understanding of "what works".

They put a reference on their site to a comparison between typical service
startup by means of a shell script and systemd.
There it is:
https://web.archive.org/web/20141020161905/http://forkfed...

I look at both ways side by side and have a sympathy for these guys.
They want to know what is *executed* by reading the script and if there is
a problem they can react almost instantly. It gives them confidence !
On the other hand, the systemd's declarative config file seems to be easier to read (even to compile), but are they able to comprehend
instantly what actually happens when those directives are processed behind the scene by systemd ? Do they have the same confidence ?
What if they want or need to make changes beyond config file ? What if they want to try something (a fix, a prototype) ? Prospect: C code files and program compilation.

There is much more to this then the above example.
I think they are right.
jb

some like it, some don't

those that don't aren't opposed to other people having the option of using systemd, but for some reason, those who like systemd seem opposed to other people not using it.

Yes - although a fork is not quite the same thing as a Debian derivative.

> > this causes serious problems for people who have some reason to need to run older kernels.
> Who in his right might will want to run such a collection of security vulnerabilities?

Someone who is hunting for performance regressions. I've seen this use case discussed in one of the past systemd threads here on lwn.

(I've also seen on one of these threads an use case of "someone who is using an embedded board which only works with an old 2.6.x kernel", but your disgust is justified in that case.)

It seems like this is the TYPICAL use case of large PostgreSQL systems.

It's also typical of RHEL and SLES systems.

As far as RHEL and SLES goes, I gather a lot of security fixes are back-ported, but enhancements typically are not. And given that a major driver of the *need* to use new kernels is new hardware (which by definition is irrelevant to most RHEL/SLES systems) that's why a lot of such systems run such old stuff. As the saying goes, "if it works, don't break it". When you're an overstretched admin running loads of systems, the last thing you want is an upgrade which breaks a production server...

Cheers,
Wol

there's also the issue that what the systemd people consider a horribly obsolete kernel is one a couple of releases back, which is new enough that it may not have made it out to any distros as their stock kernel yet.

We aren't talking about 2.6 kernels, but even 3.14 is old enough to be missing 'critical features'

I've done it to bisect kernel bugs. For example, recently I had a corrupted ext4 filesystem image which triggered an oops in the filesystem code for recent kernels but which had previously worked on kernel 3.3. In order to find the commit that broke the code, I had to use sysvinit because systemd would just refuse to work on older kernels.

Not that I would necessarily want systemd to limit itself to features of ancient kernels; just to point out that it does make some things harder.

What about the other way? Will a current kernel work with a two year old systemd?

Or stated another way, in four years time will a RHEL7/Jessie user be able to run the current upstream kernel.org kernel?

Two years is really not enough. It's even less than one Debian release, so people might legitimately have problems with that.

Perhaps 4 years should be more reasonable target? Also, can you offer official compatibility checker, so we can be sure that a given unit file would work on an older version of systemd?

> Next, the systemd project appears to be on a mission to do more than just rewrite init. It appears to be on a mission to subsume all of the core Linux userspace, either by absorbing utilities or else by rewriting them.

The thing is, a lot of these things needed work. Look at for example resolveconf. You have a number of different programs that can setup an internet connection and to decide which DNS to use you get scripts to dump files in a directory and then one is selected. I had non-working DNS for a while because some program had dumped a file there which overrode everything. Try tracking that down. This is so obviously a hack.

Now systemd has a daemon which does DNS by asking all the of the provided DNS servers, thus transparently merging all the DNS namespaces. If an interface goes down the DNS goes away, without relying on some fragile scripting. This is so obviously the correct solution I don't understand why people are complaining.

(This is actually a manifestation of the platform problem. When the given multiple DNS servers, since the glibc resolver works with resolv.conf, the answer chosen was to select one source to put in resolv.conf. Whereas the correct answer should have been, fix the resolver to handle multiple independent DNS servers).

Why do you have to run a complete NTP server daemon just to keep the system time accurate? Why would you still have cron, which can only mail output, when you have an init which support many more options? From my point of view systemd is finally providing correct solutions for problems that have been endlessly worked around.

> Service files today often contain a number of hard-coded dependencies on other services, and on things that must be run before or after the running of the service itself.
Yes, just like any other init system. Dependencies are actually NOT hardcoded - you can override them (and only them!) using custom units in /etc.

>Indeed, systemd even has directories of symlinks to encode dependencies
Incorrect. Systemd does not use symlinks to encode dependencies in any meaningful way.

> - Decrying hard-coded dependencies and sets of symlink directories, only to replace it with another set of the same but different: Hmmm...
BS.

> You are mistaken in a fairly trivial way -- systemd is far beyond being just an init system.

At its core, it's an init system. An init system, not an init daemon. Most of its extra functionality is replacing what used to be provided by the "early boot" scripts (rcS.d on Debian-style distributions), or the core parts of the "late boot" scripts (at rc[2345].d on Debian-style distributions) like filesystem mounting or networking.

Other parts of its extra functionality are things which are required for its functionality (cgroup management), which augment its functionality (like journald, which as some might remember started as the "stdio bridge" to capture a daemon's stdout/stderr output), or which work better as part of PID 1 (I believe that's the case for kdbus management).

Finally, there are the parts which just share its internal libraries of utility functions but are otherwise separate; udev is the main example (it's still separate, only the code was simplified by porting it to systemd's internal libraries), and I believe the future "userspace console" work will also be an example. These are not a concern; they are using the same git repository just for development convenience.

Other than that last category, it all can elicit the reaction "I could do that with shell scripts" (pulling things like syslog and udev from external projects, and managing them with shell scripts); in fact, that's how it was done before systemd. Thus my point stands:

> Everybody (on the Unix world) understands (or thinks can understand) how an init system works. It's something so simple that it could be implemented as a set of shell scripts (and on traditional sysvinit without an initrd, it actually is). Therefore, everyone has its own strongly-held opinion on how things should be done.

> Also, what you are looking for is "journalctl -f", I guess.

I'd say "journalctl -fa" is a better answer. Using just "journalctl -f" truncates log lines to fit the screen; "journalctl -fa" displays them untruncated and without filtering unprintable characters, just like "tail -f" would do.

> improvements in FreeBSD's init system, effectively making it more similar to systemd

There are already plenty of dependency-based boot managers. Technically systemd is nothing new, just NIH. And monolithic non-portable code can be found all the way back to the software stone age.

Nor is Embrace, Extend, and Extinguish new. Microsoft started using EEE decades ago.

FreeBSD may well implement an init system with similar functionality to systemd, but with modularity and without EEE it will not be at all similar to systemd.

You're mischaracterising people with a very broad brush here. Merely being of the opinion that systemd is badly designed and implemented does not automatically mean one is a backward luddite who won't "live with the times". This unnecessary personalisation of the discussion is one of the factors which have reduced the debate to a low level and made this sorry mess in the first place. It's one of the reasons I unsubscribed from the main lists a year back and greatly reduced my involvement (RSI was the other); I'm a volunteer who chooses to dedicate a significant portion of my life to making Debian, and being subject to continued abuse over a multi-year period is just not acceptable. I've got other free software projects I can work on instead which also provide me pleasure and accomplishment without involving disrespectful and rude behaviour and the resulting high levels of stress and upset. Looking occasionally at the list archives, the tone of the discussion has only become worse.

As one of the sysvinit/initscripts maintainers, I'd say that we were well aware that while they had served us well for decades they did have certain shortcomings and that a replacement which solved these would have been well received. We were in no way opposed to a quality replacement so long as it was backward-compatible to allow upgrades without loss of function or configuration. Three years ago, I was hopeful that systemd had the potential to be that replacement, and spent some time investigating it. For reasons which have been tirelessly debated and which I will not reiterate here, I did not ultimately feel that systemd was suitable as a replacement, and in the three years since then the course of systemd development has not changed my opinion on the matter; if anything it has further reinforced it.

The need for this fork could have been avoided entirely if certain developers were willing to accommodate continued use of the old init. But willing cooperation is not part of some developers' agendas, and that is a great shame since it is that which allowed us to build such a high quality and comprehensive integrated software distribution in the first place.
[To put this into perspective, I spent many man days of effort in the wheezy release making sysvinit align better with systemd to aid migrations and interoperability both ways; that's what you do when the goal is a well integrated distribution. I spent several solid days just making sure that hwclock timezone settings worked in every possible scenario. I think it's fair to say in retrospect that this was ultimately a fairly one-sided effort.]

While I still use Debian daily at both work and home, most of my systems have been running FreeBSD since February. I've needed to evaluate what the post-jessie migration paths might need to be. My plan is to switch over to Devuan once the dak instance and archive is set up, and participate in development at some level, though I don't know what the extent of that will be at this point. If it is small and focussed upon correcting this single major problem with Debian, I think it has a good chance of success, and we can hope that maybe the differences between the two distributions can be resolved at some point in the future. I'll continue to maintain stuff in Debian alongside this work.

Regards,
Roger

If you'll excuse the cliché, there's room for everyone at the party. Nobody's views or wishes need be excluded.

There's no need for the VUAs to "live with their times", as you put it, and likewise there's no reason for systemd proponents to not adopt systemd. The Devuan fork is not going to harm Debian, and may even be beneficial to Debian in the long term, as someone pointed out in another comment. There has already been a long history of Debian derivatives.

The majority of distributions have already switched to systemd. Surely having a selection of quality distributions which offer an alternative is a good thing? Why is choice suddenly bad?

I think there is indeed a part of the opposition having some kind of trauma with regards to Microsoft and their previous computing experiences. Reference to journald being "microsoft-like" due to binary format are common, and we can see reference at svchost.exe , conspiracy about companies trying to create a monopoly, etc.

I guess that for some people, the Microsoft hatred is a fundational brick of their free software involvement, and suddenly discovering that things are more complex than the binary manichean division of the world they had in mind shattered their views of the world, who had to be readjusted to the new changes, and readjusted using schemas of the past. They maybe are trying to make sense on what they didn't like without knowing why, being fueled by the fears of their pasts, about frustrating experiences they didn't fully grasp.

So I am not sure what they need exactly, besides people to spend time to hear them and try to understand what they really fear. The kind of customer services jobs that the free software have always been good at avoiding, by not caring and chasing people who would be good for that, by showing that only technical contributions matters, forgetting all the others.

Wow. Did you actually read jgg's post before copy-n-pasting your reply? He covers that.

> Except that NO OTHER INIT SYSTEM IS PORTABLE, EITHER !!!

systemd is not just an init system. It aspired to replace the entire plumbing layer.

Making the entire plumbing layer non-portable would have been a huge step backward.

Most other init systems are completely portable to other Unix systems because they use nothing but C and POSIX interfaces from libc. sysvinit is portable. It runs today in Debian on the Linux, FreeBSD and Hurd kernels with eglibc. It could undoubtedly run on other kernels and with other libcs such as ulibc.

The initscripts themselves do have some Linux-isms in them; primarily use of /proc which is catered for by linprocfs on FreeBSD and a translator (IIRC) on Hurd. For the scripts which are Linux-only (e.g. udev), that doesn't matter. For the others they could be cleaned up to be more portable if there was a need for that. E.g. I suspect use of /proc/mounts is no longer necessary now mtab has been eliminated; we could use the mount(8) output directly. If we wanted to remove the use of linprocfs and linsysfs, these scripts could be cleaned up just as we removed the bashisms. There hasn't been an realistic need for that, however.

I agree that the unconditional reliance on new features by systemd is a problem. You would have thought they would have the ability and resources to support conditional usage with a fallback, and to be able to test on older systems to make sure the fallbacks work.

I tried asking exactly that kind of question several times in anti-systemd camps: How do you propose to tackle the problems addressed by systemd instead?

So far the only reaction I managed to get was long arguments about whether or not that the problem exist in the first place. The issues that logind covers (simplified version: that the user on the seat -- and only him -- has the permissions necessary to use the devices connected to that seat) is one example: That issue does not exist and can be handled by user groups (apparently at the same time:-).

The fact that user groups were tried for a couple of decades before people set out to implement consoleKit, logind and consoleKit2 does not convince. Apparently the admins back then were too stupid to do group setup right and the developers that set out to implement all those services were misguided not to notice that.

That kind of reply really saddens me. I do would appreciate some competition to systemd, just to keep everybody on their toes:-) Unfortunately I could not spot any group able to pull that off so far... but maybe I was not looking hard enough.

> Bullshit, Debian didn't introduce any such dependencies, upstreams did.

That is not actually true. Except for systemd itself, most upstreams are portable and very flexible. But a few DDs are reconfiguring packages to introduce systemd dependencies.

And the GR ruled that DDs can pretty much do whatever they want to Debian.

Hence the fork.

What you're missing is that "systemd dependencies" in these people's tiny minds include any code that supports systemd socket init or startup notification. Even if that code runs fine under other init systems.

You seem to have completely missed the point of the post. If you read the parent post, you will see reasons for not using systemd on embedded systems

They are large, compleicated, and have tons of features that aren't needed for the functionality of the system.

This makes the system more fragile and vulnerable, just because it can do so much more.

Any Yes, a stripped down firewall is an embedded system. A media center computer is also an embedded system. The type or power of the hardware isn't what makes something an embedded system. What makes it an embedded system is the purpose of the system and what you can do with it. If it is setup so that you don't install anything on it, just use it as-is, it's an embedded system. If you can install apps on it, it could be an embedded system, but that depends on the apps available.

My Android Phone is not an embedded system

My Vizio TV is

My router counts and an embedded system when running the factory software, but once I install OpenWRT on it, it's classification as an embedded system becomes more questionable.

That could just be the result of a buggy distribution. If your distribution provides systemd and it doesn't work nicely, then it is ok if they say it is buggy (even if they're mistaken about the problem).

dps: Your use-case should work fine. File bugs with your distribution.

> That column reads like "justifications for being a dick" for people who failed to notice that the BOFH was a joke.

I was wondering about that, too. I read that column as one more in the genre of "how NOT to do things" and obviously tongue-in-cheek. If the project's foundational document is a joke the is the project itself a joke?

the kernel does have to deal with this sort of thing when dealing with hardware. Just because Intel released a new chip with a neat feature doesn't mean that they don't have to keep supporting old chips that lack that feature.

As a result, the kernel gets written to take advantage of the new feature, but have a fallback if that feature doesn't exist (that fallback may be to emulate the feature, or just report an error if someone tries to use that feature, but it isn't to refuse to boot without that feature)

The problem is that too many userspace developers only look at the latest. The comments about systemd supporting kernels for the last two years and consider themselves enlightened to be so good about supporting older kernels is a good example of this. While the systemd folks are congratulating themselves for being so good, other people are looking at this and saying that this is way too short a timeframe, that two years is barely enough to get a kernel from kernel.org into the hands of users in many environments.

Userspace developers who have to deal with large installed bases of third party stuff tend to be much better about this. So Samba, *syslog, browsers, webservers, e-mail software, etc all tend to look for the latest to take advantage of it, but have fallbacks to keep working in the absence of these features.

It's mainly in the Desktop Environment space that the developers seem so willing to expect that everyone will upgrade everything at once and nobody has any business running anything older.

>You're are not listening to the admins and users. Most of us don't want systemd.

Please speak only for yourself. None of us have any real idea what most users or admins want. We just know that major distributions are shipping it by default.

I am a former pro sysadmin, a current amateur sysadmin, and a current developer. I've written my own init scripts and suffered.

I had to learn a few new things but I like systemd rather a lot. I know others who also like it. We are VERY HAPPY with it and therefore we don't bother complaining about distros using it.