This whole debate saddens me

Posted Nov 30, 2014 3:01 UTC (Sun) by dlang (guest, #313)
In reply to: This whole debate saddens me by mpr22
Parent article: The "Devuan" Debian fork

putting it another way, why do those things need to be reimplemented inside systemd rather than continuing to be init (and kernel) agnostic?

This whole debate saddens me

Posted Nov 30, 2014 5:03 UTC (Sun) by mchapman (subscriber, #66589) [Link] (27 responses)

> putting it another way, why do those things need to be reimplemented inside systemd rather than continuing to be init (and kernel) agnostic?

resolved and networkd *are*, as far as I can tell, init-agnostic. networkd makes use of hostnamed's public API in order to set the system hostname. I think that's literally the only thing that links these utilities with the other systemd components.

As for why they should be in the same tree? One good reason is that they can share code easily. A large part of systemd is its "shared" directory, which acts as a source library that is linked into its various binaries. This provides a degree of consistency across the various systemd components.

This whole debate saddens me

Posted Dec 2, 2014 12:10 UTC (Tue) by nix (subscriber, #2304) [Link] (26 responses)

As for why they should be in the same tree? One good reason is that they can share code easily. A large part of systemd is its "shared" directory, which acts as a source library that is linked into its various binaries. This provides a degree of consistency across the various systemd components.

So... almost all this stupid, tiresome systemd arguing could have been avoided if the systemd people had created a shared library (like anyone else would have) and set up more than one git repo? You'd think that would have been the default course of action anyway, but maybe they're scared of library API design. (Surely not: the internal library has an API just as surely.)

So, I am at a loss...

This whole debate saddens me

Posted Dec 2, 2014 12:56 UTC (Tue) by mchapman (subscriber, #66589) [Link] (5 responses)

> So... almost all this stupid, tiresome systemd arguing could have been avoided if the systemd people had created a shared library (like anyone else would have) and set up more than one git repo?

Possibly, possibly not. It might be worthwhile considering what the trade-offs there are. Distributions are already shipping networkd and resolved in separate packages from systemd-the-init, so there has to be a compelling reason why you'd add an additional shared library for these components to use.

One advantage to having everything in the one Git tree is that code can be moved in and out of the shared directory at will. If you have things in separate trees then this becomes a lot more difficult -- suddenly you have to *version* your internal APIs, not just your external ones.

Also, just in case you're wondering, while there is some overhead in having multiple binaries linking the same set of shared source files, it isn't too much. systemd is compiled with function section (or with link-time optimisation), which means unused code is completely dropped. Each binary gets only the shared functions it actually *uses*.

This whole debate saddens me

Posted Dec 2, 2014 17:00 UTC (Tue) by johannbg (guest, #65743) [Link] (4 responses)

"Distributions are already shipping networkd and resolved in separate packages from systemd-the-init"

Which distribution are that?

This whole debate saddens me

Posted Dec 3, 2014 0:17 UTC (Wed) by mchapman (subscriber, #66589) [Link] (3 responses)

> Which distribution are that?

Ah, you got me.

I was getting confused. I actually had a particular Fedora COPR repository for RHEL 7 in mind: https://copr.fedoraproject.org/coprs/lnykryn/systemd/ . It's currently got networkd in a subpackage. resolved is currently disabled, though as far as I can see there shouldn't be anything stopping it from being put in a subpackage.

This whole debate saddens me

Posted Dec 3, 2014 13:56 UTC (Wed) by johannbg (guest, #65743) [Link] (2 responses)

Zbyszek decides how he indents to maintain systemd in Fedora until someone escalates the packagesplit ( which at is point,after all this time would be a bit silly from my pov ) to FESCo, which in turn would most likely request input from BaseWG ( which probably would go for the smallest core/base footprint which means not split it ) followed by FESCo rule based upon that and his maintainership being overruled and then each WG overule the overule and decide for themselves if they want to split it ( or not ).

Lukáš RHEL 7 and derivatives corp repo that you refer to is only intended to be testing repository of new systemd releases ( I think he's already drowning in backporting patches for the release being shipped in RHEL 7 with RHEL 7 being bound to be stuck on that release for the next 10 years or so ).

This whole debate saddens me

Posted Dec 3, 2014 23:48 UTC (Wed) by mchapman (subscriber, #66589) [Link] (1 responses)

Sure, but that's all kind of irrelevant to this thread.

The context of my earlier post was about the modularisation of systemd, and whether having a shared library for its internal APIs (rather than a source library) would be worth it. The existence of this repository indicates that a shared library isn't even necessary to package systemd's components separately.

This whole debate saddens me

Posted Dec 4, 2014 0:39 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

True. Somebody who is interested in it could talk to Fedora maintainer for example and submit a spec file patch. It shouldn't too hard.

This whole debate saddens me

Posted Dec 2, 2014 13:39 UTC (Tue) by cesarb (subscriber, #6266) [Link] (19 responses)

> So... almost all this stupid, tiresome systemd arguing could have been avoided if the systemd people had created a shared library (like anyone else would have) and set up more than one git repo?

They use a single repository for the same reason the Linux kernel is developed as one single, monolithic repository: it makes it *much* easier to make tree-wide changes which impact shared code.

> Surely not: the internal library has an API just as surely.

Of course it has. But, like the Linux kernel internal API, it's fluid and can change from one commit to the other. As long as the corresponding changes are made to all the API consumers at the same time (easy when your version control system has atomic commits), nothing will break.

This whole debate saddens me

Posted Dec 2, 2014 17:58 UTC (Tue) by dlang (guest, #313) [Link] (1 responses)

so if the systemd API between components is "fluid and can change from one commit to the other" then it's not really something that anyone is free to replace components of because there is not stability in what the component is supposed to do.

That's a monolithic entity (like the kernel is), which systemd claims it isn't.

This whole debate saddens me

Posted Dec 2, 2014 18:07 UTC (Tue) by niner (subscriber, #26151) [Link]

You are mixing APIs between components and the API of the common library these components currently use. The API between components can be perfectly stable and public while the library can change any time.

Exactly the same as the API the Linux kernel offers to user space is absolutely stable while internally everything is fluid.

This whole debate saddens me

Posted Dec 10, 2014 17:55 UTC (Wed) by nix (subscriber, #2304) [Link] (16 responses)

Of course it has. But, like the Linux kernel internal API, it's fluid and can change from one commit to the other.

To me that's a sign of developer laziness: they can't be bothered to design APIs right the first time, and they're making everyone else pay for that. (It *can* be done, though it is a lot harder than just guessing and constantly chopping and changing them.)

This whole debate saddens me

Posted Dec 10, 2014 18:41 UTC (Wed) by peter-b (subscriber, #66996) [Link]

>> Of course it has. But, like the Linux kernel internal API, it's fluid and can change from one commit to the other.

> To me that's a sign of developer laziness: they can't be bothered to design APIs right the first time, and they're making everyone else pay for that. (It *can* be done, though it is a lot harder than just guessing and constantly chopping and changing them.)

Designing good, future-proof APIs is extremely difficult, as you acknowledge. Maintaining APIs in a bug-compatible manner is much more difficult still. It's totally reasonable for developers to define a small, stable, external API, while maintaining flexibility within their software to modify internal implementation details. It enables improvements to the efficiency, reliability, or features of the implementation. In fact, this approach is evidence of a robust development process and a sustainable project.

By contrast, it is completely *unreasonable* to expect people to only ever design, write and release perfect, flawless, future-proof code, and accuse people who have realistic expectations of what is achievable of "laziness". It is, in fact, arrogant and insulting.

What's going on here is that the systemd developers defined some public APIs that they wanted to provide, and went ahead and implemented a system that provided those APIs. You're now nitpicking their implementation. systemd's public APIs are fully documented, with stability guarantees, and that makes the project a heck of a lot more professionally developed than 99.9% of FOSS software projects.

I invite you to: 1. "improve" systemd to match your expectations (or pay someone else to), 2. fork systemd and "fix" it to your satisfaction, or 3. produce a plumbing suite that solves the same key problems in a "better" way. Until then, perhaps you should consider *not* wantonly slinging dismissive insults around. They're not going to get you whatever it is that you think that you want.

This whole debate saddens me

Posted Dec 11, 2014 0:02 UTC (Thu) by mchapman (subscriber, #66589) [Link] (14 responses)

> To me that's a sign of developer laziness: they can't be bothered to design APIs right the first time, and they're making everyone else pay for that.

Did you completely miss the word "internal"?

These APIs are *not* public APIs. They are *not* used by other software. They are merely the internal function calls from one part of a binary to another. They aren't even visible to or accessible by end-users.

This whole debate saddens me

Posted Dec 11, 2014 0:41 UTC (Thu) by dlang (guest, #313) [Link] (13 responses)

but the systemd folks keep claiming that systemd isn't monolithic, that anyone is free to replace any component, they just need to implement the same functions.

but if the API of those functions is "internal" then it is monolithic in design (even if implemented in multiple binaries)

This whole debate saddens me

Posted Dec 11, 2014 0:49 UTC (Thu) by pizza (subscriber, #46) [Link]

> but if the API of those functions is "internal" then it is monolithic in design (even if implemented in multiple binaries)

The API *between* components is a published, stable DBUS (and/or command-line) interface.

The implementation details, ie the C functions, are internal, private. You know, not unlike Linux's published userspace API, vs the internal implementation details that are changing all the time?

This whole debate saddens me

Posted Dec 11, 2014 3:11 UTC (Thu) by mchapman (subscriber, #66589) [Link] (11 responses)

> but the systemd folks keep claiming that systemd isn't monolithic, that anyone is free to replace any component, they just need to implement the same functions.

Oh criminy. Let's take this from the top again.

You asked why the different systemd components are in the one tree. I said that one likely reason is so that they can share code. nix wondered why they didn't just use a shared library for that. I said that it might be because the shared code isn't part of the *public* interface of systemd, so there is no compelling reason to make it a dynamically-linkable library.

Got it yet? We're talking about code that is statically linked into multiple, separate, independently-reimplementable binaries. This has *absolutely nothing* to do with their public APIs (i.e. a set of D-Bus interfaces) provided by those binaries.

This whole debate saddens me

Posted Dec 11, 2014 17:22 UTC (Thu) by nix (subscriber, #2304) [Link] (10 responses)

Quite. I happen to think this is shoddy design, unless (like, say, gnulib) there are very good reasons for it (in gnulib's case, that a lot of the horrifically evil dark magic it does happens before compile time, at the configure stage, and that it's only trying to fix up holes in already-public APIs anyway). It was a bad idea when GNOME did it with its 'cut-n-paste-code' directories, it was a bad idea when people embedded dozens of copies of libpng and zlib, and it's a bad idea now. Yes, doing it right is more work. Doing it right often is.

And no, I'm not going to go away and fork or reimplement systemd or shut up as peter-b so charmingly suggested. I don't have infinite time. I am, in fact, permitted to say 'this is bad design, other projects don't do this, it has been done this way before and every time turned out to be a bad idea' without simultaneously being obligated to fix every single instance of bad design I see. Nobody could ever have enough time to do that.

This is a classic systemd proponents' attack: if you criticise the holy project in any way you should shut up unless you have personally implemented something better. Sorry, it doesn't work that way, and repeatedly attempting to shut down any and all criticism this way does *not* make me feel better about the project as a whole.

This whole debate saddens me

Posted Dec 11, 2014 19:30 UTC (Thu) by peter-b (subscriber, #66996) [Link] (4 responses)

Well, you can think it's shoddy design all you like. But that's your opinion. It's completely unreasonable to expect any software project to go to the trouble of maintaining bug-compatible stability of internal interfaces that are explicitly labeled as not for public use.

Show me a free software project that has such a ridiculous policy.

This whole debate saddens me

Posted Dec 13, 2014 2:49 UTC (Sat) by flussence (guest, #85566) [Link] (2 responses)

Does Wine count?

This whole debate saddens me

Posted Dec 13, 2014 3:30 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link]

Not really. They had no choice given the purpose of their project.

This whole debate saddens me

Posted Dec 13, 2014 12:39 UTC (Sat) by cesarb (subscriber, #6266) [Link]

> > But that's your opinion. It's completely unreasonable to expect any software project to go to the trouble of maintaining bug-compatible stability of internal interfaces that are explicitly labeled as not for public use.
> >
> > Show me a free software project that has such a ridiculous policy.

> Does Wine count?

Wine's main internal, not-for-public-use interface is the server protocol, which is the RPC between the lower-level Wine DLLs and the shared wineserver process.

They don't maintain bug-compatible stability of that interface. Every time they feel the need, they add, remove, or change its methods and structures. The only concession to compatibility is a single version number, which is increased ever time they make a change, and used to detect version mismatches between the server and the DLL. That version number is already over 450, meaning there were over four hundred times that interface was changed in an incompatible manner.

So no, Wine doesn't count.

This whole debate saddens me

Posted Dec 14, 2014 17:21 UTC (Sun) by nix (subscriber, #2304) [Link]

Projects at the same layer: glibc. libgcc. libstdc++. Projects at a different layer: kdelibs (yes, really, right down to internal functions). libX11.

There is a certain degree of conservatism that should come with writing software at the kernel/userspace boundary, or at the 'everyone has lots of dependencies on me' level. glibc has it. systemd (like libpng) doesn't.

This whole debate saddens me

Posted Dec 11, 2014 23:49 UTC (Thu) by mchapman (subscriber, #66589) [Link] (3 responses)

> I happen to think this is shoddy design

This the first time I've heard of "calling a function in a different compilation unit" shoddy design. If that's what you really think, then all software is shoddy.

This whole debate saddens me

Posted Dec 14, 2014 17:24 UTC (Sun) by nix (subscriber, #2304) [Link] (2 responses)

It's not shoddy design unless you then turn around and say that things that use said functions in those compilation units are not really part of that software at all, they're just bundled into the same source tree so they can use that library interface! That position implies very strongly that that interface is actually an external interface, since things can call it that are not 'really' part of the project.

As far as can see, there are two possibilities here:

- Either those interfaces are external interfaces, and they should be a stable shared library API, and the systemd developers are doing shoddy work by not making them stable, or

- those interfaces are internal interfaces, subject to change as and when needed, and those things in the systemd source tree that use them are *part of systemd* and claims that oh no they just share a source tree and are not really part of it at all are, not to put too fine a point on it, lies.

I don't see a third option, and neither of these are particularly pleasant.

This whole debate saddens me

Posted Dec 14, 2014 20:15 UTC (Sun) by anselm (subscriber, #2796) [Link] (1 responses)

I don't know whether explaining this to you once more will make any difference or if you're deliberately trying to be obtuse, but here goes:

systemd as a whole consists of a number of modules that communicate via documented external interfaces. There is a stability promise that covers most of these interfaces. It is possible to replace most of those modules by independent implementations that are otherwise unrelated to the systemd code base as long as one sticks to the documented interfaces.

In the interest of saving work, various modules avail themselves of common subroutines in a separate directory in the source tree. Since all of systemd is released together and nothing outside of systemd uses these subroutines, there is little point in making these into a shared library (especially since then invariably somebody outside the project would want to to use that, and that would impact the systemd developers' freedom to change things around as needed to suit the implementation), and systemd uses linker features to ensure that every systemd modules contains only the code it actually uses.

Therefore systemd's "library API" is an internal implementation detail similar to the internal APIs of the Linux kernel. It only exists to make the systemd developers' life easier. Whether systemd consists of separate modules (it does) is of no consequence to this, because it is still possible to reimplement many of these separate systemd modules independently based on the documented external interfaces and the API stability promise.

This whole debate saddens me

Posted Dec 14, 2014 21:05 UTC (Sun) by deepfire (guest, #26138) [Link]

It should be a FAQ entry somewhere. You've written it up quite well..

This whole debate saddens me

Posted Dec 13, 2014 0:29 UTC (Sat) by njs (guest, #40338) [Link]

I don't see the comparison to the cut-and-paste code or libpng/zlib issues. The problems there are that you end up with multiple copies of the same source code, which then diverge.

In the systemd case, there's exactly one copy of the source code, and it always gets updated in sync with its reverse-dependencies. You do end up with multiple copies of the *compiled* code, but I don't see how this can cause problems beyond a miniscule waste of space. It's not really any different from how macros or inline functions or C++ templates work. Plus it almost certainly increases reliability to avoid adding another .so dependency to your early boot stack, esp. once you take into the account the possibility of version skew etc.

This whole debate saddens me

Posted Nov 30, 2014 18:23 UTC (Sun) by anselm (subscriber, #2796) [Link]

They don't "need" to be implemented in systemd, and you don't need to run them even if your system is otherwise based on systemd. Nothing prevents you from running, say, NetworkManager if systemd-networkd doesn't float your boat.

It just makes reasonable technical sense to offer these tools in conjunction with systemd, as a common baseline for systemd-based systems. They get to share a lot of the basic infrastructure of systemd, which means that the code in question needs to be implemented, debugged, and documented only once rather than over and over again. Remember also that one considerable advantage of systemd is that it unifies configuration methods across different Linux distributions, and distributions would otherwise have to implement and maintain their own versions of basic tools for network configuration and so on. Getting these for free from systemd is not just good for consistency but also frees distribution resources for work on other things.

As far as cron is concerned, in the traditional setup it is part of a whole zoo of different tools whose collective job it is to start processes given certain conditions (this zoo also includes the rc mechanism, at, inetd/xinetd, and the init process itself via /etc/inittab, among other things). Considering that systemd's main business as an init system is starting processes given certain conditions, and that it tends to do it a lot better than the traditional zoo used to and with a consistent, well-designed configuration interface, it does make sense to let systemd do cron's job because it contains all the required infrastructure already. This saves us from having to maintain the same sort of infrastructure over again as part of cron, especially since systemd does all sorts of useful things that cron doesn't support and, being essentially in a coma upstream, is unlikely to get anytime soon. In addition, it's not as if it were impossible to generate the requisite systemd units on the fly from existing crontabs, so you even get to keep your traditional configuration if you want to.