I talked with Lennart in person about this. He was saying is that they'd discussed this in systemd but never implemented anything related to it.

One specific interesting topic that was discussed is having a protocol between the bootloader and the runtime where the "boot count" is stored as a suffix on the filename. Something like /boot/loader/entries/ostree-fedora-0.conf.5 where the .5 is new and represents "number of times to try this entry".

On success, drop the suffix to just /boot/loader/entries/ostree-fedora-0.conf. On failure, decrement the count. If the count reaches 0, skip to the next entry.

And he'd considered having a generic boot-success.target or so in systemd, but not a specific implementation. It's presumably something define downstream in Fedora - potentially having a different one in each variant.

thanks @walters for the summary. I think your first comment applies more to #3.

For your second comment, yes. Here are the discussed next steps for an inital POC for @lorbus:

• define boot-success.target systemd target
• create systemd unit that may fail (and should be easy to trigger failure for testing)
• If boot-success.target is not reached use OnFailure directive to start some other process (initially just echo FAILURE to a file)

After those are done I think we can start to look at #3.

First, rather trivial and naive stab at it:

https://github.com/LorbusChris/greenboot

I'll also ask for tipps and feedback on the systemd mailing list tomorrow.

@lorbus I have some very minor feedback on your RPM spec. https://github.com/LorbusChris/greenboot/blob/master/greenboot.spec#L6-L17 needs some formatting love. When I write my own specs I normally use two spaces for tabs and I try to keep things aligned. It makes it easier to read and it lets other's help you better as a result.

Here is an example of one of my own specs:

https://copr-be.cloud.fedoraproject.org/results/jdoss/wireguard/fedora-28-x86_64/00758394-wireguard-tools/wireguard-tools.spec

Maybe you are already aware of these resources, but I will post them anyways. When writing RPM specs I always seem to end back at http://ftp.rpm.org/max-rpm/ which will everything you would ever want to know about creating RPMs. Other good resources I have found are:

https://rpm-packaging-guide.github.io/
https://docs.fedoraproject.org/quick-docs/en-US/creating-rpm-packages.html

Hope this helps and nice first stab. Keep poking!

Thanks @jdoss!
It did look aligned in my editor and I also saw how screwy it looked on GH, but couldn't get that fixed quickly so I let it be. And after all that's merely cosmetics. I'll try with 2-space tabs!

Here's what I wrote on the systemd-devel mailing list:
https://lists.freedesktop.org/archives/systemd-devel/2018-May/040796.html

migrated comment from @jlebon:

OK, played around with greenboot for a bit. (I really like the name BTW!)

I think I understand why you went with a service instead of a target -- though OTOH, I really like the semantics of targets for our use case. systemd has these nice well-known targets which serve as interfaces for other services to hook into. One I recently came across was: the nss-user-lookup.target can be used to separate services that provide NSS lookup services (like SSSD) from those that require it (like systemd-logind). Similarly, a greenboot.target can be the interface other units plug into. (E.g. both for running things before the target, like the various tests, but also after, e.g. telling the mothership we're ready).

Here's a strawman:

• greenboot.target with a WantedBy on multi-user.target
• test services can put a RequiredBy/WantedByongreenboot.target`
• and then a greenboot.service with After=greenboot.target to run whatever we want if we had a successful boot or not? (I.e. use systemd API to get the status of greenboot.target and e.g. update the bootloader config, stop the watchdog, etc...)

migrated comment from @lorbus:

@jlebon: Thanks for the feedback! I'm seeing the usecase for targets now more and more, too and had a similar idea of what to use greenboot.target for (see my comment in issue #3)

I'll incorporate this in the next iteration of greenboot!

Hi there, re the bootloader -> systemd protocol for tracking a number of boot attempts:

We're doing something similar on our embedded system. We've got two read-only rootfs partitions and two R/W config partitions. These are called slot-A (rootfs-A + cfg-A) and slot-B. At any given time, just one of these partitions is active while the other one is inactive. When a system is booted from, say, slot-A, any update is installed into slot-B, and the bootloader is set to perform the next boot from slot-B. There is a number of attempts to try for each slot, and once it gets too low, the next slot becomes active again. This is a pretty common system, and there was a bunch of talks from various FLOSS projects and companies on how they are solving this.

We're currently using RAUC for updating the entire system. Everything is integrated with our bootloader (U-Boot), the numbers are tracked in the U-Boot's env. We've also verified that this works with grub2, and the RAUC project also integrates well with Barebox' bootchooser (Barebox is another bootloader). For a reference, check our patch to U-Boot which ensures that stuff works.

There are also other projects which target a similar use case, for example swupdate. People from both swupdate and RAUC gave a number of talks about where they are coming from and what problems they solve. My recommendation is to get in touch with them. For example, some systems might not have space for two copies of the rootfs, others might want to have a special, never-updated rescue slot, etc.

One thing that I was disappointed with is systemd's support for watchdog. Basically, my bootloader sets up a HW watchdog which reliably reboots the CPU unless the kernel periodically keeps telling it "hey, everything is OK". This is typically done via systemd, but there's a nasty bug in there because systemd happily keeps hitting the watchdog saying "hey, I'm up" even if it fails to reach its target. For more details, see [systemd-devel] Later activation of the HW watchdog which unfortunately didn't go anywhere. A TL;DR version of that is "I want systemd to only start touching the watchdog once it successfully reaches its default target".

If you're interested in this, feel free to reach me at jan.kundrat@cesnet.cz . I don't have (human) bandwidth for tracking this thread :), and we're using Buildroot, not Fedora's IoT for our builds anyway.

With kind regards,
Jan

Coming from HA background, this idea looks appealing also in that area,
especially if the implementation/underlying checks will stay highly
machine-specific, user-driven and the framework as generic as currently
presented with greenboot.

While there are some inherent health measurements performed as part
of the cluster operation, it may make no sense to try to bring the cluster
stack up (join the cluster partition) should be the basic preconditions of
successful participation unmet (no network interfaces, HW/SW self
evaluations indicating issues, etc.).

I am not very versed in systemd graph transitions, but it could be possible
to also provide redboot.target mutually conflicting with greenboot.target
and setting the latter to OnFailure=redboot.target. That would then
allow say pacemaker.service as Wants=greenboot.target and
Conflicts=redboot.target, which would then in this case prevent
pacemaker from coming up when the precondition is not satisfied.

I see this may be somewhat orthogonal to the original use case here,
but providing enough flexibility for other use cases (while standardizing
on the "custom checks" interface) might be a worthy goal.

@jpokorny thank you for your thoughts on this!
I believe this is a valid usecase and merits further discussion!

For ostree-based systems, we'll want to roll back to a previous, working (as in successfully booting) version of the OS. However, this is AFAIK not applicable for most other OSes/distros.

Abstracting away the case-of-failure into its own target thus sounds compelling to me.
On ostree-based systems, what should the behaviour be if there is no previous version to be booted back into?
WDYT @jlebon @walters @dustymabe @pbrobinson?

(btw special thanks to @jlebon and @walters for their recent, very helpful feedback on greenboot, via various channels :)

On ostree-based systems, what should the behaviour be if there is no previous version to be booted back into?

This should probably be configurable. Some configurable options include:

• (default) just fail a systemd unit with a big error message and maybe update the motd
• option to tell the system to shut itself down (might be useful in cloud providers so another node will get brought up in the cluster by a cluster master
• attempt boot again (maybe next time it works for some reason)

Following up on my idea, couldn't greenboot.servicebe eventually
removed from the dependency graph, hooking every (non)criticial test
just to greenboot.target and then possibly having rpm-ostree-redboot.service
oneshot service with

After=greenboot.target
Conflicts=greenboot.target

(pacemaker example above should also be extended with After=greenboot.target
I think)?

Naive idea here is that systemd would try to satisfy greenboot.target first,
and if it's evaluation (and hence of the critical tests) was successful, nothing
would happen (rpm-ostree-redboot.service would fail), otherwise
rpm-ostree rollback; reboot would be started (which could be moreover
emulated by returning nonzero and having FailureAction=reboot)

If a scheme like this actually works (cannot attest), it would have these benefits:

• decoupled check and response parts (assuming rpm-ostree-redboot.service
  would either be enabled only in OSTree context or part of a separate package
  not strictly required by greenboot)

• no scratching of systemd surface from within systemd-based handler
  when the same logic can be emulated natively with mere transition
  graph execution

Should we close this issue and move all discussions over to https://github.com/LorbusChris/greenboot/issues ?

Agreed, the official upstream for this project is:
https://github.com/LorbusChris/greenboot/

Package will be in Fedora soon (under review) and the usual mechanisms for Fedora will be in place.