Conspect: Why are we adding connections between pieces of software at widely different levels? - the graphical user session talks to the boot loader to set boot-loader flags (systemd-bless-boot, grub-boot-sucess) - the system manager mounts and unmounts user directories (systemd-homed) - user directories are moved onto the main system partition (btrfs) - systemd units are used for user-level applications - systemd is taking over the management of gnome session - systemd generators manage desktop unit autostart (xdg-autostart-generator) - desktop services talk to the kernel memory manager (uresourced) - userspace services take memory management away from the kernel (earlyoom, systemd-oomd) Instead of having nicely separated layers we are moving towards a system where everything seems to know about everything (and systemd has an ever-growing contact surface). All components are becoming more complicated and entangled. Where is the process going, what is the motivation, are there any gains, will this ever stop? I want to discuss the developments in the list above, show that they make sense and are not as dangerous as some people say, listen to even more ideas about integration of components. Talk: Let's talk about some changes that have been happening recently in Fedora https://in.waw.pl/~zbyszek/why-does-my-bootloader-care-about-gnome-terminal.txt Disclaimer: systemd-centric and gnome-centric The graphical env stuff is cross-desktop and cross-distro. From the bottom → the changes at the bottom are actually easier to understand: the implementation is often complicated, but the goal is well defined "Boot blessing" - the boot loader cannot know if the selected kernel actually works - we let the userspace *decide* when a boot is successful and *inform* the boot loader - the boot loader can take action We have two implementations: in grub and sd-boot grub: boot_indeterminate++ /usr/lib/systemd/system/grub-boot-indeterminate.service /usr/lib/systemd/user/grub-boot-success.{timer,service}: mark the boot successful in grub when a user manager has been running for two minutes The effect is to show the boot menu if the boot fails at least once. https://fedoraproject.org/wiki/Changes/HiddenGrubMenu https://fedoraproject.org/wiki/Changes/FlickerFreeBoot https://hansdegoede.livejournal.com/19081.html systemd: /usr/lib/systemd/system/systemd-bless-boot.service: mark the boot successful in sd-boot (systemd-bless-boot-generator: enable system-bless-boot.service when sd-boot is active) boot-complete.target after boot-complete.target: systemd-bless-boot.service before boot-complete.target: ??? systemd-boot-check-no-failures.service → marked as "proof of concept" in the documentation Let's consider a failure mode: kernel crashes in early boot [grub-2.02/util/grub.d/10_reset_boot_success.in] → OK, we mark the boot as failed, and we never mark it as successful Another failure mode: user logs in, but the mouse and keyboard input freezes → the session runs, we mark the boot as successful :( The lesson from all this: the high level components "know" what is happening. But the high level shouldn't care or have privs to talk to the low level. We need generic scaffolding that allows things to plug in. sd-boot allows failed boot counting. To make this useful for something like tablets the whole scheme would need to be much more robust. Systemd units for the user session Clarification: *user* manager units, not *system manager* / pid1 units How does this work: - gnome-shell uses org.freedesktop.systemd1.StartTransientUnit() to let systemd user instance manage the processes - by itself, this doesn't change much. Various internal bits need to be reworked in glib. [https://gitlab.gnome.org/GNOME/glib/-/merge_requests/1596] [https://gitlab.gnome.org/GNOME/glib/-/merge_requests/1592] - systemd has the generic infrastructure to manage groups of processes using cgroups. $ systemd-cgls /user.slice/user-1000.slice Motivation (3 items): 1. systemd is good at keeping track of processes 2. the whole job dependency graph can be used for the graphical session The graphical shell can do what it good at (the graphical user-interfacy stuff). xdg-autostart-generator → convert ~/.config/autostart/*.desktop into systemd user session units 3. we can assign resources to cgroups Side track: how the kernel manages resources for cgroups: cpu, memory, io weights limits (hard, soft) protections (low, min) allocations [https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#resource-distribution-models] We can set all those settings for systemd units, and systemd takes care of configuring the appropriate kernel knobs. The kernel is good doing the low-level resource allocations, but it has no idea which processes are important. F33: uresourced https://fedoraproject.org/wiki/Changes/Reserve_resources_for_active_user_WS Essentially sets memory.low, cpu.weight, io.weight *for the active session* Why can't the kernel do this? The kernel doesn't know what is a "foreground session" and doesn't know which processes belong to it. More controversial: F32: earlyoomd https://fedoraproject.org/wiki/Changes/EnableEarlyoom "It is the job of the linux 'oom killer' to sacrifice one or more processes in order to free up memory for the system when all else fails." → the userspace knows which processes are important to the user and is able to set a policy that is suitable for an interactive user session Common theme: High-level userspace components have knowledge and are the place where user interaction happens. We need a fairly complicated middle layer (systemd, systemd --user) to help the high-level user space communicate with lower layers. The lowest layers are where monitoring and execution actually happens. (mechanism, not policy) Reasons: → accounting can be only done efficiently in the kernel → policy updates are slow, and the kernel sometimes needs to act fast. We cannot rely on a userspace component in critical moments. → the kernel is always there, the machine can't "hang" Is this all in violation of the "Unix principles"? The Linux system as a collection of tidbits that the admin can put together in any way is gone. Conceivably, the kernel could do this, but it would need to hardcode all the settings. Systems are becoming more and more heterogeneous. The number of connections up and down the stack is becoming ever greater. Note: many things were always possible. The difference is in how this is all integrated. We are not writing a script which reaches deep into the bowels of something else. We finally generic interfaces which can be put together in a scalable way. Similar story to the boot entry blessing — the highest layers talk indirectly to the very low ones. We need a lot of plumbing. Some developments in progress: Companies are making use of resource management in the kernel. On desktops, we're still taking baby steps. earlyoomd, uresourced → systemd-oomd, systemd-logind PSI – live (!) information about "pressure" Instead of saying "this much is used" the kernel says "the lack of *this* resource is slowing things down" $ cat /sys/fs/cgroup/user.slice/user-1000.slice/cpu.pressure some avg10=1.00 avg60=1.02 avg300=1.06 total=490134812 $ cat /sys/fs/cgroup/user.slice/user-1000.slice/io.pressure some avg10=0.00 avg60=0.00 avg300=0.00 total=33525775 full avg10=0.00 avg60=0.00 avg300=0.00 total=27898839 $ /sys/fs/cgroup/user.slice/user-1000.slice/memory.pressure some avg10=0.00 avg60=0.00 avg300=0.00 total=2975025 full avg10=0.00 avg60=0.00 avg300=0.00 total=2238598 This is hierarchical – we can monitor individual services. We execute a task, and we can see that for 30 s during the build, 20 processes were waiting for CPU, and for 10 minutes during the build, 5 processes were stalling on IO. This allows use to replace the static earlyoomd policy of "kill things when the system is running out of memory" with "kill this background processing task when user might even notice a slowdown" "protect this task at all costs" "freeze this task when there's memory pressure" "freeze this background task when user logs in?" Resource management is hierarchical — and the hierarchy needs to be managed. The old approach of "this kernel compilation is underneath the graphical terminal which is underneath the dbus daemon which is underneath the ssh session which is underneath the sshd daemon (because the user happened to log in over ssh first)" just doesn't cut it. If the hierarchy is logical and not based on internal implementation details, users can understand it. If the hierarchy is similar between gnome, kde, sway, … we can write generic policies. If the hierarchy is organized by function, we can distribute resources in a functional way. Future developments: - cross-desktop cooperation - systemd --user session finally starts being utilized - even more integration: select the appropriate kernel boot loader entry from a graphical menu? manage cgroup resource allocations for tasks? make the graphical sessions more robust to partial failure?